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SUMMARY 

OBJECTIVES 

The promises of Individualized materials and computer-based In- 
structional systems for Improving training effectiveness and efficiency 
have been recognized by the Department of Defense as viable means for 
reducing military technical training costs. In the military training 
environment, every day of training time saved by innovations in train- 
ing materials or procedures can result in considerable savings in train- 
ing dollars and more efficient utilization of manpower in the field. To 
date, however, little effort has been devoted to the problem of pre- 
paring students to use computer-based or computer-managed Instructional 
(CM I) systems effectively. 

The goals of the CMI Student Skills Project were, therefore, to (a) 
determine the characteristic problems which students encounter in a 
computer-managed instructional (CMI) system and those strategies which 
effectively help students cope with these problems, (b) design, develop, 
implement, and evaluate a small set of self-contained instructional 
modules for increasing the effectiveness with which students adapt to 
and perform in a CMI environment, and (c) investigate procedures for 
individualizing assignment of these modules so as to minimize training 
time and cost. The project context was the Air Force Advanced Instruc- 
tional System (AIS), Lowry AFB, Colorado. 

APPROACH 

The student skills modules developed were designed as short 
packages to be assigned near the beginning of an arbitrary military 
technical training course, but which also Incorporated strategies or 
procedures that would continue to affect student behavior throughout the 
course. In the process of module design and implementation, it became 
apparent that Instructor understanding and appreciation of project goals 
and objectives, as well as implementation procedures and strategies in- 
volved in the skill modules, were critical to the potential success of 
such skill training. For this reason, an Instructor orientation and 
training package was developed in place of the originally planned 
computer-based individualization procedures. 

METHODS 

To determine the characteristic problems which students encounter 
in a novel CMI environment, students from two AIS courses were inter- 
viewed in two phases of the project. Interviews in the first phase 
focused on student reactions to the CMI environment, features they liked 
and disliked, problems they had experienced, and information they would 
have found helpful about the environment. An Orientation to CMI Module 
and a Time Management Module were designed on the basis of these initial 
student Interviews. 
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The second phase of student interviews focused on specific student 
study skills problems in a CMI environment. The results of these inter- 
views, along with information from instructors and supervisors, formecl 
the basis for the design of a Study Skills Package. This package con- 
tained a Student Study Skills Questionnaire and separate skill training 
materials in the areas of Reading Comprehension, Memorization, Test 
Taking, and Concentration Management skills. Additionally, and Instruc- 
tor Orientation and Training Workshop was designed to (a) acquaint in- 
structors with the techniques and strategies in the four Study Skills 
Modules, (b) teach them basic diagnostic and tutorial skills for the in- 
dividualized assignment of students to the Study Skills Modules, and (c) 
provide them with a better perception of their CMI instructor role. 

All skills modules, the Study Skills Questionnaire, and the in- 
structor package were subjected to formative evaluation, and revisions 
were made as appropriate. Due to operational constraints, summative 
evaluations (a second set of large scale tryouts) were completed for 
only the Time Management Module (using students from the Inventory 
Management (IM) and Materiel Facilities (MF) courses) and the Study 
Skills Questionnaire (using students from the IM and Weapons Mechanic 
(WM) courses). 

RESULTS AND CONCLUSIONS 

Operational tryout of the Orientation Module was restricted to the 
Precision Measuring Equipment (PME) course. During the evaluation 
period, the module was administered to students in the first CMI block 
of the course in an Orientation/No-Orientation evaluation design. No 
reliable differences were found between the block times, block scores, 
or attitudes toward Cfll for students in the two groups. These results 
are tentative due to low student flow and, hence, small sample sizes. 
The average time spent by students on the module was only i\2 minutes. 
Indicating tnat tne time required for such an orientation Is not exces- 
sive. 

The Time Management Module was subjected to an extensive two-phase 
summative evaluation, during which (a) the module and the AIS Student 
Progress Management Component (SPMC) were .ompared with a no-module, no- 
SPMC condition in Phase I and (b) the moouie per se was evaluated in the 
presence of the SPMC, as compared with a no-module, SPMC condition in 
Phase II. The results of these evaluations revealed time savings of 
11. <^ and 9.U percent, respectively, with the overall effectiveness of 
the module and SPMC conservatively estimated to be in excess of 18.0 
percent by the end of the Phase II evaluation. It was concluded that 
the time spent on the module (approximately 80 minutes) was a cost that 
was quickly amortized in terms of the savings in technical training 
time. 

The four Study Skills Modules were subjected to an operational try- 
out in the IM, MF, and WM courses. Although the number of cases for 
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this evaluation was small (n«11), the findings of dramatic student 
Improvement In block times and scores following the study skills 
remediation were so consistent as to warrant the conclusion that this 
training met the goal of Increasing student efficiency and effectiveness 
in a CMI environment. 

The Study Skills Questionnaire was subjected to a full summatlve 
evaluation In which data pn questionnaire reliability and validity were 
collected In the four AIS courses. The results Indicated that the 
questionnaire and its subscales had good reliability and construct 
validity. The predictive validity analyses supported the power of this 
measure to discriminate between students who would perform well versus 
those who would perform poorly in a CMI environment such as the AIS. 

Evaluation data for the Instructor Orientation and Training Work- 
shops were provided by instructor critiques of the workshops and by sub- 
sequent instructor use of their new skills in assigning students to 
particular study skills materials. Results were favorable in both 
areas and it was concluded, therefore, that the Instructor Orientation 
and Training was successful in promoting the remediation of student 
study skills problems— a finding further substantiated by the consistent 
improvements in student block times and scores following this reme- 
diation. In addition, the training appeared to have a positive effect 
on Instructor role perceptions. 

RECOMMENDATIONS 

The overall results of this project indicate that it is not only 
feasible but also beneficial, in terms of training efficiency, effec- 
tiveness, and cost, to train students in the basic skills required to 
perform effectively in a CMI environment. On the basis of these 
results, it is suggested that the present student skills modules at 
least be implemented in the AIS courses. It is also reconmended that 
th** Study Skills Package be given trial implementatlcn in the non-CMI 
courses where these same basic s^"'ls would be impcTtant. Implementa- 
tion should follow procedures disc-jsed in this report, with special 
emphasis given to the role of the instructor. 
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SECTION I: EXECUTIVE REPORT SECTION 



The benefits to be derived from computer-managed Instruction (CMI), 
within the framework of large-scale military technical training, are 
very promising. CMI Is an Individualized Instructional system In which 
the majority of the students' Instructional activities are completed 
off-line, In contrast to computer-assisted Instruction (CAI) where all 
Instructional activities are conducted on-line at an Interactive 
computer terminal. The computer's role in CMI Is that of evaluator, 
diagnostician, prescrlber, and manager of Instructional events. Al- 
though considerable effort has been devoted to Improving the hardwar'^, 
software, and Instructional technology which support CMI systems, the 
problem of preparing students to utilize their skills effectively and 
efficiently within this Individualized Instructional system has 
received little attention. 

It must be assumed, moreover, that until various forms of Individ- 
ualized Instruction become common in our public school system, military 
trainees will find CMI to be an extremely novel learning experience. 
Few of these trainees will possess the knowledge or skills to enable 
them to use the capabilities of computer-based systems efficiently. 
Although there are certainly some basic skills which transfer from one 
learning environment to another, many trainees either will not have 
these skills or will not know how to adapt them to CMI training. If 
the CMI systems being designed and built are to be most effective, there 
Is a definite requirement for orienting students to novel system capa- 
bilities and for equipping them with minimum skills to capitalize on 
these capabilities. 

The goals of the CMI Study Skills Project were therefore to: 

1. Determine the typical problems which students face In a CMI 
system and those methods which help students cope with these 
probl ems . 

2. Design, develop. Implement, and evaluate a small set of In- 
structional materials for creasing the effectiveness with 
which students adapt to and perform In a CMI environment. 

3. Design these materials to be as Independent from course 
material as possible such "that they could be assigned near the 
beginning of an arbitrary technical training course. 

4. Incorporate strategies or procedures that woul'd continue to 
affect student behavior throughout the course, e.g., review 
and practice of a set of basic skills. 

Proiec t Context; The Air Force Advanced Instructional System . 
The context for the CMI Student Skills Modules Project was tne Air Force 
Advanced Instructional System (AIS), develop^d^^iinder the auspices of 
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the Air Fore* Human Resources Laboratory Technical Training Division 
and located at Lowry AFB. The AIS H a prototype, multi-media, computer- 
based Instructional system designed to Improve the effectiveness and 
efficiency of Air Force technical training and to provide an operational 
research facility for assessing Innovations In Instructional technology. 
The system supports four technical training courses representative of 
the range of cognitive and performance skills required by enlisted Air 
Force personnel. An adaptive Instructional decision model utilizes 
state-of-the-art computer hardware and software, as well as currently 
available statistical methodologies and Instructional procedures, to 
provide Instructional management and Individualized assignments to 
alternative Instructional materials. 

AIS Course Structure. Fich AIS course Is divided Into "blocks" of 
instruction which may require anywhere from 1 to 15 days to complete. 
Each block contains a number of lessons and a comprehensive, end-of- 
block test. Within a block, lessons are arranged In a hierarchy based 
on their prerequisite relationships. A typical hierarchy resembles a 
set of parallel chains diverging and converging on certain pivotal 
lessons, and a student may alternately work on lessons In two or more 
parallel chains. 

The basic unit of Instruction Is the lesson. Each lesson consists 
of a set of objectives, two or more forms of a criterion test, and 
typically, a self- test by which the student can evaluate his or her 
understanding of the lesson before taking the criterion test. A 
lesson's Instruction Is provided by one or more modules, each of which 
teach the same lesson objectives and cover the same lesson content. 
Where two or more modules are present, they represent alternative in- 
structional treatments or strategies. Depending on the lesson content, 
objectives, and nature of the treatment, a module may be a programned 
text, an elaborated technical order, or an audio-visual presentation. 

AIS Student Progress Management Component . The AIS Student Pro- 
gress ManagementCoi^^ target course completion 
time for each student. These targets are predicted on the basis of the 
student's abilities, attitudes. Interests and background information as 
measured by a battery of tests given all students prior to course entry. 

The initial SPMC printout occurs when the target rate is first 
computed, following the processing of the student's last preassessment 
test form. Target times for each block and the student's total cour«>e 
completion time are listed in units of days and tenths of days. This 
printout is delivered to the student's learning center Instructor, and 
the information on this printout is used by the student in completing 
the progress monitoring procedures described in the Orientation/Time 
Management module. 

The student's first prescription or Status Report of each day con- 
tains the days and tenths of days of the course completed and the days 



and tenths of days spent In class. The amount of work constituting a 
"day" of the course is a function of the student's target rate. Each 
student's rate of progress is also reported on the learning center 
roster which instructors receive at the beginning of each shift. Infor- 
mation on the roster includes the number of days and tenths of days r'e- 
maining to the student's targeted course completion date and the number 
of days and tenths of days by which the student is ahead of target. A 
negative value for "days ahead of target" Indicates that the student is 
behind target and provides a means by which instructors, can detect 
students who are falling behind in their course work. 

If an instructor decides that a student's target rate should be re- 
set, a target change request can be forwarded to the course Database 
Manager, who changes student targets through an interactive editor. 
Although it would have been feasible to alter students' target rates 
automatically on the basis of their actual rates of progress, specific 
intervention by the instructor was purposefully required. Given the 
variety of reasons why students may be behind or ahead of their target 
rates, it was reasoned that the instructor is in a better position to 
determine the correct action than is the SPMC software. 

PROCEDURES 

To determine the typical problems which students face in 9 CMI 
environment, students from two AIS courses were interviewed In two 
phases of the project. During the first phase, students from subgroups 
defined as "good" and "poor" in each course were asked general questions 
about (a) their reactions to the CMI environment, (b) features they 
liked and disliked about this environment, (c) problems they had exper- 
ienced, and (d) information about that environment which they would have 
found helpful as new students. "Good" students were those who were com- 
pleting the course faster than the average student with above average 
grades. "Poor" students were those who were completing the course at a 
slower than average rate with below average grades. All selected 
students were working in the last half of their training course and were 
selected on the basis of records maintained by the AIS. An Orientation 
to CMI Module and a Time Management Module were designed on the basis 
of these initial student interview results. 

The second phase of student interviews was conducted after the 
design and development of the Orientation and Time Management Modules. 
These interviews focused on specific student study skills problems in a 
CMI environment. For these interviews, students were selected from sub- 
groups defined as "Experienced Good," "Experienced Poor," "Naive Good," 
and "Naive Poor." "Good" students were those who were at least 2 days 
ahead of their Target Completion Date and had at least an 80% grade 
point average on completed block tests. "Poor" students were those who 
were 2 days or more behind thdir Target Completion Date and had less 
than an 80% grade point average. "Experienced" students were those who 
had completed all but two blocks of their course, and "naive" students 
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were those who were still working on the first block of their course. 

The results of these student Interviews, along with Interviews with 
students who had been eliminated from the course and Interviews with In- 
structors and supervisors, formed the basis for the design of a Student 
Study Skills Package. This package contained a student Study Skills 
Questionnaire and separate skill training In the areas of Reading Com- 
prehension, Memorization, Test Taking, and Concentration Management 
skills. Additionally, an Instructor Orientation and Training Workshop 
was designed to help Instructors learn and practice the techniques and 
strategies In the four Study Skills Modules and to update their basic 
diagnostic and tutorial skills so that they would know how to Identify 
students who needed help In the study skills area. 

ORIENTATION MODULE 

The major design goals for this njcdule were to (a) Improve 
students' attitudes toward CMI and (b) reduce course completion time. 
It was hypothesized that If stjdents were given a solid understanding 
of novel CMI environment features and Instructional procedures, they 
would have a better appreciation for the benefits of this Instructional 
system and would waste less time trying to figure the system out for 
themselves. It was also felt that students would begin using the CMI- 
provlded Instructional tools earlier In the course and would, accord- 
ingly, spend more time learning the Instructional materials. A sub-, 
ordinate goal for this module was to provide students with appropriate 
role models for Increasing acceptance of their Increased responsibility 
for learning in a CMI environment. 

The Orientation Module can be described as a brief and general 
introductory package which explains the characteristics of a typical 
CMI environment. It compares and contrasts these characteristics with 
those of a conventional lock-step environment and details the behaviors 
of successful versus unsuccessful CMI students. The module emphasizes 
that in a CMI environment the student is responsible for learning and 
that many times negative self- talk is the root of student problems. 
Finally, this module introduces the concept of time management and its 
Importance in terms of achieving one's goals. 

The first evaluation of the Orientation Module was done in a small 
group tryout during which students and staff read the module and were 
then Interviewed on a one-to-one basis to determine their opinions and 
suggestions. The ""esults of these tryouts Included (a) the mean time 
to complete the module was approximately 20 minutes, with a range of 
16.1 to 23.5 minutes, (b) students expressed a need for embedded 
questions to be added to the module, (c) students and instructors 
requested changes to several of the cartoons, and (d) a number of 
explanations of efficient student behaviors for CMI were in need of 
clarification. Based on these results, appropriate revisions to the 
Orientation Module were made prior to operational trycuts. 
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The second evaluation of the Orientation Module was restricted to 
the PME cciurse, since evaluations of the Study Skills Modules were 
scheduled 'in the other three AIS courses. For approximately 6 weeks, 
data were collected on block completion tlines and scores and pre/post 
Attitude Toward CMI scores, using data from students who entered and 
completed :he second CMI block in the course during this period. To 
provide an .evaluation of the relative effectiveness of an Orientation 
versus No-Orientation condition, students were randomly assigned to 
Module 01 (Jr 02, respectively, for this first lesson of the block. 

At the completion of the evaluation period, the data showed only 
nine cases for Module 01 and seven cases for fiodule 02. When time 
constraints were applied to exclude unreasonable cases (excessive block 
completion times), the number of cases for Module 01 dropped to four 
and the number of cases for Module 02 dropped to six. Analyses on both 
of these seV,s of data revealed no reliable differences between the 
groups on block times, block scores, or attitude measures. These 
results, although somewhat inconclusive due to the small number of 
students available for this evaluation, are promising in light of 
anecdotal data supplied by both students and instructors. That is, 
conments concerning this module indicated that (a) there was a de.inite 
requirement for orienting st-dents to the novel aspects of a CMI environ- 
ment and (b) the module design was positively received. Furthermore, 
shortly after this evaluation was completed, the PME course transitioned 
their entire course to a CMI format and installed the Orientation 
Module at the beginning of the first block of the course. 

TIME MANAGEMENT MODULE 

Time Management Module . The major design goal for this module was 
reduction in course completion time beyond any reduction attributable 
to the AIS SPMC. Subordinate goals were that (a) students would main- 
tain their individual progress tracking charts on a daily basis, (b) 
students and instructors would meet for scheduled Progress Counseling 
Sessions, and (c) students would express positive attitudes toward CMI 
in general and the SPMC in particular. 

The content of the Time Management Module focused on explaining 
the philosophy, characteristics, and operation of the SPMC and the use 
of the Course Completion Map. The Course Completion Map is a progress 
charting technique for promoting stlf-monitoring and self-management 
student behaviors in a CMI environment. 

The first evaluation of the Time Management Module was conducted 
in the Inventory Management (IM) course and began with two small -group 
tryouts in which tht main concern was with the mechanics of the module 
itself. The evaluator worked one-on-one with a total of 42 entering 
students. In addition to the module, students were given a lesson test 
and a five-item attitude questionnaire. As a result of these tryouts, a 
number of minor wording changes were made and additional expUnations 
and examples of daily progress plotting were added. 
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This was followed by a second evaluation of the Time Management 
Module. During this evaluation, the Time Management Module became the 
first lesson in the first block of a special evaluation version of the 
IM course. The module questionnaire was not administered since a stan- 
dard Student Attitude Questionnaire was administered at the end of Block 
1. During this evaluation, 64 students entered the evaluation vgrslon . 
of the course, and of these, 28 completed the first four blocks .^he 
number completing the remaining two blocks of the course was too small 
for meaningful analysis. 

First attempt scores on the lesson test had an average of 88.3 
percent. Only one student failed to meet the criterion of 60 percent 
correct. First attempt lesson times were found to be unreliable due to • 
the lesson's position as the first assignment in the course. Adminis- 
trative activities required at course entry were being charged against 
the lesson, and the lesson's position was later changed to avoid these 
timing errors. 

Students' completion times and end-of-block test scores on the 
first four blocks were compared with those of a comparable group of 
students entering the course during the same period. The results were 
considered promising since the average of the evaluation group's total 
time for the four blocks was 3.75 hours less than that of the control 
group. 

The Time Management Module was then evaluated in a third test. 
During the first phase of this test, the module, plus the AIS SPMC were 
compared with a no-module, no-progress-managament condition. During 
the second phase, all students were under the SPMC but only half of 
them were given the module. 

Results of these evaluations revealed time savings of 11.2 percent 
in Phase I and 9.0 percent in Phase II for the module. The overall 
effectiveness of both the module and Management Component were conser- 
vatively estimated to be in excess of 18.0 percent by the end of the 
Phase II evaluation. In addition. Phase II results, unlike Phase I 
results, indicated nr negative block scores, block failures, or student 
attitude results with the combination of Time Management Module and SPMC. 

STUDY SKILLS MODULES 
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V The major design goal of the Study Skills Modules was to provide 
poorly skilled students with appropriate techniques for improving their 
performance in the course and reduce the amount of time required to com- 
plete the course. The four study skills areas which were isolated as 
being the most troublesome for students were (a) Reading Comprehension, 
(b) Memorization, (c) Test Taking, and (d) Concentration Management. 
A set of materials was written for each of these four areas and pack- 
aged individually, so that a student could receive any or all of the 
packages as deemed necessary by an instructor. The modules were alsu 



designed to be consumable (I.e., retained by students) so that students 
could return to new material to freshen their memories or clarify con- 
fusing or difficult areas. Furthermore, since the use of good study 
skills Is an on-going requirement In the course. It was judged desirable 
to give students a set of materials to which they could refer whenever 
necessary. 

The philosophy behind the techniques described In all four of the 
Study Skills Modules Is that learning new Information becomes easier If 
the student changes the new Information In such a way as to make It 
meaningful to him or her. Thus, the Reading Comprehension Module recom- 
mends that students ask questions about the new mater* il they are trying 
to learn or draw pictures detailing the relationships between the new 
ideas. The third recommendation of this module is for the student to 
use a systematic problem-solving procedure to resolve difficult or con- 
fusing passages. The Memorization Module describes and exemplifies the 
use of mnemonics and the Test Wiseness Module discusses the use of 
logical reasoning strategies and gives students numerous practice 
exercises in these techniques. The Concentration Management Module 
discusses the Importance of creating a good mood for efficient and 
effective study and the ways in which negative self-talk can cause 
numerous problems. 

The first evaluation of the Study Skills Modules employed instruc- 
tors from the four AIS courses, rather then students. There were three 
main reasons for this procedure. First, It was hypothesized that if 
Instructors were included in the formative evaluation of this package, 
they would be more accepting of it during subsequent evaluations. Such 
acceptance was deemed highly desirable due to the substantial Instructor 
resistance experienced during evaluation of the Orientation/Time Manage- 
ment Module. Second, since training in study skill techniques was to be 
an Integral part of the Instructor Orientation and Training, it was 
thought that instruf :or participation would be greater if their comments 
and suggestions were given validity and value. Finally, much of the 
material which was used in these modules had already been subjected to 
student evaluations by other researchers. 

The results of this evaluation indicated that the instructors 
generally liked the modules and believed that they would be useful to 
at least some of their students. They also thought, however, that the 
modules were written at too difficult a level for most students and, 
therefore, needed to be revised. 

Instructor Orientation and Training was conducted prior to the 
second evaluation of the Study Skills Modules in order to insure that 
(a) only those instructors who were committed to the utility of study 
skills remediation participated in this evaluation and (b) those same 
Instructors were adequately trained to use the materials appropriately. 
Given that Instructor volunteers for these workshops were only obtained 
from operational AIS blocks in the IM, MF, and WM courses, operational 
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tryouts of the Study Skills Modules were restricted to these courses. 
The evaluation period was approximately 10 weeks. During this time, 
instructors identified students in need of one or more of the Study 
Skills Modules and assigned them to these modules during the regular 
shift or assigned the module(s) as homework. To assist instructors in 
their diagnostic activities, a special AIS interactive program was 
written for them. This program, when given a student's Social Security 
Number, displays the student's Study Skills Questionnaire scores. . 

The results of the second evaluation of the Study Skills Modules 
were very encouraging. Although the number of cases for this evaluation 
was small (instructors forgot to record student data in the AIS data 
base or they did not attend the instructor workshops and therefore did 
not understand the importance of this activity), the findings of 
dramatic student improvement in block times and scores following study 
skills remediation were so consistent as to warrant the conclusion that 
this training met the goal of increasing student efficiency and effec- 
tiveness in a CMI environment. There was also some evidence that pro- 
viding students with study skills training improved their opinions of 
their study skills, particularly if the'training was given in more than 
one study skills area. Finally, the results suggested that the time 
required for study skills remediation compares favorably with the 
resulting savings in completion time (around a 17 percent reduction in 
course completion times for those students receiving the Study Skills 
Nodules). 

STUDY SKILLS QUESTIONNAIRE 

The Study Skills Questionnaire is divided into four sections corre- 
sponding to the four Study Skills Modules: Reading Comprehension, 
Memorization, Concentration Management, and Test Wiseness. For the 
purposes of this project, the questionnaire was administered twice: 
after students completed the first block of their course and just before 
beginning the last block. The results of the first administration of 
the questionnaire provided a pre-intervention measure and information 
for instructors as to the type of remediation that would be most 
appropriate for specific students, and the results of the second admin- 
istration of the questionnaire provided a post-intervention and eval- 
uation measure. 

The first evaluation of the Study Skills Questionnaire consisted of 
instructor reviews of this instrument. The results of this review 
indicated that instructors were generally sati'sf^ed with the content and 
format of the questionnaire. The questiorinaire was then evalu-.ted a 
second time and as such was given to, all students in the IM, MF, WM, 
and PME courses as they began the second block of their CMI course and 
again as they began the last block of their course. Analysis of these 
data indicated that 30 of the original 5U questions were highly reliable 
and should be used in the revised questionnaire. 



The Study Skills Questionnaire was then subjected to a third eval- 
uation in which data on both the reliability and validity of the 
questionnaire were collected in the four AIS courses. The results of 
this evaluation indicated that the questionnaire was highly reliable 
and able to accurately and consistently predict whether or not students 
would perform poorly in their course. Performance was measured by block 
completion times, blpck scores, lesson completion times, and lesson 
scores. In addition, although the smaller set of Study Skills Question- 
naire variables was not as effective as the larger preassessment variable 
sets in correctly classifying unsatisfactorily versus satisfactorily 
performing student groups, the results were encouraging for those CMI 
environments which do not accommodate preassessment testing. For an 
environment such as the AIS which does accommodate a preassessment 
battery, the Study Skills Questionnaire adds another useful dimension of 
individual difference data. 

INSTRUCTOR ORIENTATION. ANa' TRAINING , 

The Instructor Orientation and Training Workshop has three main 
objectives: (a") To familiarize instruG^ors with the strategies, tech- 
niques, and objectives of the study skill materials, (b) To provide 
instructors an opportunity to practice these techniques, and (c) To 
provide instructors with appropriate counseling and tutorial skills. 
This third objective was deemed important, as one of the major respon- 
sibilities of a CMI instructor is considered to be providing one-to-one 
counseling for students with special learning problems. 

The Instructors Workshops consist of three 2-hour sessions. The 
first session is devoted to explaining, discussing, and practicing the 
use of the study techniques described in the Study Skills Modules. The 
second session focuses on problem solving skills, diagnostic strategies, 
and remediation procedures, including some training in listening and 
probing skills. These two sessions are held on consecutive days and the 
third session is held approximatel^y 1 week later. This third session is 
designed to give instructors a chance to discuss any problems or ^ 
difficulties they have experienced in using the materials and procedures 
in their classrooms, to exchange information, solutions, and suggestions 
and to present interesting case histories. 

Evaluation data-for the Instructor Orientation and Training Work- 
shop were provided by both the instructor critiques of the workshop, and 
by instructors' subsequent use of their new skills in the assignment of 
students to particular study skills materials. Instructor comments from 
the workshops were generally very favorable, with a majority of the 2/ 
participating instructors indicating that they liked the content and 
format of the workshops. 

Unfortunately, however, not all of these instructors had the 
opportunity to assign study skills materials to students in their courses 
due to changes in their duty assignments following the workshop tiain- 
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ing. Of the nine instructors in the IM, MF, and WM courses who were 
able to assign the material, an average of more than two students per 
instructor were given Study Skills Module(s) assignments. Judging by 
discussions with instructors and data clerks in these courses, however, 
these numbers are misleading in that many more study skills assignments 
were made but were not recorded in the AIS data base. Thus, it was con- 
cluded that the Instructor Orientation and Training was at least 
moderately successful in promoting the remediation of study skills 
problems— a finding further substantiated by the consistent Improvements 
in student block times and scores following this remediation. 

SUMMARY 

Results and Conclusions from Student Skill Modules Project 

° The data on the Orientation flodule were inconclusive, but 
anecdotal data indicate the module is of benefit to students. 

° The Time Management Module, when Combined with the SPMC, resulted 
in significant reductions in block and course completion times. 

° The Study Skills Questionnaire was highly reliable and a good 
predictor of student performance. 

° The data on the Study Skills Modules, although limited with 
respect to number of samples, point to dramatic and consistent 
student improvement in -block times and scores following assign- 
ment to study skills remediation. 

° The Instructor Orientation and Training Workshop was well liked 
by participating instructors and used by them in the classroom 
for the assignment of students to study skills remediation. 

° Some evidence was found that the Instructor Orientation and 
Training led to improvements in instructor attitudes toward the 
CMI system and their new roles in that system. 

IMPLICATIONS FOR FURTHER RESEARCH 

This project, has demonstrated the positive benefit of student skill 
training on reducing the costs of military technical training. There 
are, however, a number of questions that remain unanswered. This 
section lists these questions as areas reconmended for future research. 

1. The need exists to investigate the er* recti veness of the Orien- 
tation Module in the AIS and/or other CMI technical training environ- 
ments, with respect to its impact on students' performance and attitudes. 

2. Research aimed at individualizing the assignment and/or re- 
assignment of time management skill training is desirable, in that it 



has the potential of further increasing student training efficiency and 
effectiveness. 

3. As with student training in time management skills, there is a 
need to investigate methods for individualizing the assignment of study ^ 
skills training in order to obtain maximum benefit from this type of V 
training. 

4. Additional research which pin-points the cut-off scores on the 
Study Skills Questionnaire which are most reliably related to study 
performance (times and scores) in CMI technical, training courses, as 
well as to their need for particular types of study skills remediation, 
is needed to. further the utility of this questionnaire. 

5. A critical need exists to explore the types of roles required 
of instructors in a CMI environment, particularly as these relate to 
their function as facilitators of the learning process, and to provide 
specialized CMI instructor role training. 

RECOMMENDATIONS FOR USE OF MATERIALS PRODUCED IN THIS PROJECT 

1. The Orientation/Time Management Modules should be implemented 
near the beginning of CMI technical training courses to improve student 
efficiency and attitudes. 

2. The Study Skills Questionnaire should be made part of each 
course's preassessment battery or placed in the first course block, so 
that it can be used to help Identify students with specific study skills 
problems or those who will have difficulty successfully completing the 
course. 

3. The four Study Skills Modules should at least be implemented in 
all CMI courses and used by students identified as having study skills 
problems. Consideration should also be given to using these skill 
modules in non-CMI Technical Training courses. 

4. An Instructor Orientation and Training Workshop in those skills 
required to effectively use the Study Skills Questionnaire and Modules 
should become an on-going in-service training program in each course. 
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SECTION II; SCIENTIFIC DATA SECTION 
1.0 I NTRODUCTION AND OVERVIEW 
OF CMI SKILL MODULES PROJECT 



Recent innovations in instructional technology offer substantial 
promise for improving the effectiveness and efficiency of military train- 
ing. Advances in individualized instructional materials and procedures 
fULn!'*®''^^!^^ noteworthy in this regard, particularly those which take 
advantage of recent concepts enprging from the field of cognitive 
psychology and its focus on the active information processing capa- 
bilities of the learner and strategies for differentially Improving the 
effect veness of these unique processing skills. At the simplest level 
of individualization, instructional materials can be designed to allow 
the student to complete the instruction at his or her own pace. The 
addition of computer-based instructional procedures enhances the individ- 
ualization benefits of self-paced instruction by providing the infor- 
mation management capabilities required to implement more adaptive 
Individualized Instruction on a large scale. 

The promises of individualized instruct.nal materials and computer- 
based instructional systems for improving training effectiveness and 
efficiency have been recognized by the Department of Defense as viable 
means for reducing military technical training costs. Perhaps in no 
other instructional environment is it more apparent that time equals 
dollars. Every day of training time saved by innovations in training 
materials or procedures can. result in considerable savings of training 
dollars and more efficient utilization of manpower in the field. 

In the context of large-scale military technical training, benefits 
to be derived from compiitfr-managed instruction (CM!) are particularly 
promising. CMI can be defined as an instructional system in which the 
majority of the students' instructional activities are completed off- 
line, in contrast to computer-assisted instruction (CAI) where all 
instructional activities are conducted on-line at an interactive com- 
puter terminal. The role of the computer in CMI is that of evaluator, 
diagnostician, prescriber, and manager of instructional events. Although 
considerable effort has been devoted to improving the hardware, software, 
and instructional technology which supports computer-managed instrjc- 
tlonal systems, the problem of preparing students to utilize thel" skills 
effectively and efficiently within this system has received little 
attention. 

While the problem of helping students transition their existing 
skills to innovctive instructional environments is not unique to military 
technical training, a computer-nanaged, individualized military training 
environment provides a rich arena for exploring materials and procedures 
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designed to ease the students* transition Into this new training exper- 
ience. It must be assumed that until various forms of Individualized 
Instruction become common In our public school system, military train- 
ees will find CMI to be an extremely novel learning environment. Few 
of these trainees will possess the knowledge or skills which enable 
them to use the capabilities of computer-based systems efficiently. 
Although there are certainly some basic skills which transfer from one 
learning environment to another, many trainees will either not have 
these skills or will not know how to adapt them to computer-managed 
training. If the CMI systems being designed and built are to be most 
effective, there Is a definite requirement for orienting students to 
novel system capabilities and equipping them with minimum skills to 
capitalize on these capabilities. 

1.1 Project Goals 

The overall goals of the project were to (a) determine the charac- 
teristic problems which students encounter In a CMI system and those 
strategies which effectively help students cope with or adapt to these 
problems, (b) design, develop. Implement, and evaluate a small set of 
self-contained Instructional modules for Increasing the effectiveness 
with which students adapt to and perform In a CMI environment, and [c) 
Investigate procedures for Individualizing the assignment of these 
modules so as to minimize total completion times and training costs. 

The student skill modules developed had the design goal of being 
short packages which could be assigned near the beginning of an arbitrary 
technical training course, but which also would incorporate strategies 
or procedures that would continue to affect student behavior throughout 
the course (i.e., behavioral self-control strategies). Thus, the 
rationale governing module design was to include those instructional 
strategies and procedures appropriate not only to the teaching of 
specific ski-lls, but also to the review and practice of a set of basic 
skills defined as necessary for effective and efficient student perform- 
ance in a CMI system. 

In the process of module design and implementation, it became 
apparent that additional mechanisms were necessary to effectively trans- 
ition the student skill modules into the CMI environment. Instructor 
understanding and appreciation of the goals and object ves of the pro- 
ject, as well as implementation procedures and strategies involved in 
the student skill modules were judged to be critical to the Potential 
success of such skill training. For this reason, t was decided to 
develop an instructor orientation and training package to acquaint CMI 
instructors with project goals and with the basic diagnostic and tutor al 
skills required for the individualized assignment of student study skills 
training iterials. This instructor package was. therefoi^. develop in 
place of :the originally planned computer-based individualization pro- 
cedures . 
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1.2 Related Literat>ire 

The literature related « the skill trainirg materials developed in 
this project is selectively >wed in four sections. The first three 
sections address factors in th a problem areas that are considered to be 
important in the design of materials: (a) orienting students to novel 
learning environments; (b) nroviding students with skills for managing 
their time in self-paced instructional systems; and (c) providing 
students with specific study skills required in individualized, self- 
paced training environments. The fourth' section addresses the problem 
of providing instructor training in skills which will facilitate 
effective student use of the skill /"aining materials. 

1«2.1 Student Orientation to Novel Learning Environments . On the 
basis of the literature describing military (Fletcher, 1975; Hansen, 
Ross, Bowman, & Thurmond, Note 1; Kimberlin, Note 2; McCombs, et al.. 
Note 3, McCombs & Siering, Note 4; Middleton, Papetti, & MIcheli, 1974) 
and non-military (Alien, Meleca, & Myers, Note 5; Cooley & Glaser, 1969; 
Countermine & Singh, 1974; Danford, 1974; Hagerty, Note 6; King, 1975; 
Ullery, 1971) CMI environments, the factors that are most novel to the 
student may usefully be categorized along three dimensions: (a) a 
physical dimension involving the student's interaction with the physical 
aspects of the environment, (b) a learning process dimension including 
those training features which, either through design or accident, have 
a direct ir)act on the student's rate of learning, and (c) a social 
dimension involving those interpersonal dynamics directly related to the 
preceding factors. 

Novelties the student must adapt to in the physical environment 
include (a) a variety of multi-media materials, (b) learning centers con- 
taining 10 to 100 student carrels, (c) a variety of carrel designs from 
individual to multi-person, which may contain sophisticated equipment, 
(d) instructor stations or resource centers for obtaining learning 
materials, (e) testing rooms equipped with reader/ printer terminals 
and/or interactive terminals, and (f) mark-sense answer sheets for test- 
ing and requesting student assignments. 

In the le«\rning process dimension, the student's new experiences 
may include (a) issignment of a variety of instructional materials on 
the basis of the student's characteristics or performance, (b) avail- 
ability of organizers such as objectives, embedded questions, spaced and 
massed reviews, (c) frequent criterion referenced testing, (d) individ- 
ualized pacing, (e) computer scheduling of learning activities and 
equipment, and (f) unanticipated equipment or computer failures which 
interrupt the learning process. 

The novelties inherent in the social dimension can include (a) less 
opportunity to discuss course content with peers, since students are at 
varying points in the course, (b) less opportunity to assess one's own 
performance relative to others due to the absence of group, norm- 
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referenced testing, (c) more emphasis on self-responsibility for learn- 
ing rather than on instructor or peer group, (d) objective (computer) 
performance evaluations rather than subjective (instructor) evaluations, 
(e) individual interactions with instructors rather than group- 
instructor interactions, and (f) computer-assigned seating patterns 
rather than patterns based on peer relationships. 

In her review of the impact of computer-based ins^njction on 
education and training. King (1975) notes that negatfV- student att tudes 
are one set of outcomes that can be modified by systematic orientation to 
the computer-based system. In fact. King suggests that such a-J orien- 
tation is the most feasible way to elicit initially positive attitudes 
which are unlikely to change. An orientation program should, according 
to King, provide an overview of system components and instructional 
modes to be encountered, avoid overselling the system by presenting a 
realisitic picture of the system's positive and limiting features, and 
help students overcome feelings of "machine shyness" by addressing 
common misconceptions about computer capabilities. 

Kopstein and Seidel (1972) discuss factors to be considered in 
attempting to remove the perceived dehumanizing aspects a computer- 
based system. These factors include changing students' ini^Jal percep- 
tions from a state of "can't do" to "can do." giviuy students the feeling 
that they have a choice in the management of their earning activit es. 
and helping students learn that the system is adapt ve to their abilities 
and needs. Further, they suggest that materials which have a degree of 
warmth and sensitivity to student needs are one of the "«st critical In 
gredients in altering the perception of dehumanization. -Je successful 
implementation of such materials, with respect to more positive student 
attitudes with a computer-based versus traditional math program, is 
reported by Smith and Hess (Note 7). 

Additional factors to be considered in modifying potentially 
neqative attitudes toward a computer-based instructional system are 
Sted by Khan and Weiss (1973). Their recommendation is that various 
forms of persuasive conmuni cation be used to help students feel they have 
some choice and positive benefits associated with their learning 
activities, such as increased feelings of responsibility and determin- 
ation of their degree of success in obtaining course goals, in a 
similar vein. Seidel. Wagner. Rosenblatt. Hillelsohn. and Stelzer (1975) 
llSaest that students should be provided initial training in self- 
Sement%\inrJo increase their responsibility for their own learning 
in independent learning environments. 

In a quasi -longitudinal study on the relationships between academic 
achievement and personality characteristics. Kifer 0?75) points out 
that instructional systems which provide for mastery learning and self- 
Dacinq can be instrumental in the development of positive personality 
TharaaerUtid in students. Kifer argues that the provision of positive 
features in the lea^-ning environment, such as mastery learning and self- 
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pacing, provides a means for students to achieve well th..c n^nm«*4 

Although the literature selectively reviewed in thiQ co/.fi«« 
replete with suggestions for the type iZZtC s e hou be 
given.ln an orientation to a novel learning environment th.rp ?? i • 

IZTffTA h«3e ^ys :^a Ua iy'^ es g ted 

sJ!H»„f^?"^rr! orientation packages with respect ti changes ?n 
students attitudes or perfonrance. This. Is particularly true of 

enSiJo'^L^?!"""' '° """""'^ "ov^rfe^ur^s'Sra-jHI 

_ 1.2.2 Time Hanaqe ment Skill Training . Several of the studies 

'? "^"'^'"^ paragraphs stressed the n^ortan^ of sllf- 

rtTg"!/^:ii\^Sens^r ;$rsei57":ra-?"i,isf^"5ii&- 

i?™ wnllent k?n. In' h"^-; approaches to tSng 

^nTL. ^!^K J n e<'ucational/psycholog1cal literature are 

sparse at best. One reason for this dearth of research is ths trln? 
tional concern^ With level of achievement ratSerlhan Uar ing tfles 
- When emphasis is placed on student efficiency in an individualized in- 
structional situation, time management ski ll^raining becomes ™re 

i„.i.JIl!/"? 'raining in time management skills in an individualized 
instructional system is suggested by Carpenter (1971) She lists tSi- 
following as desired outcomes of individualized -nstr ctio^- a) 

le ;ro"Si:?s*"r"'".r3 'r?''' o^'"™ ty ""and d fflcuUy 
levels Of materials and methods, (c optimum adaptation to student 

{I 'oSHm-^sl^iJfrr "'""^ of student act? U?es. ,nd 

IciiSit es Yilhtlrrh"'"" *° self-regulated learning 

aciivities. To obtain these outcomes, a module for trainina ^t„Amt^ in 

time management skills for a self-pac^d. comp ter-ba e" train ng e v rin- 

S sh ft Jr^'elt^r^^r^n'';"/?" ^"i':*^"*!^ Pac*-? oneself and 1 i™g 
ine sniTt from external control to self-regulated learning. 

A number of recent studies have investigated the differential 
effectiveness of student-paced versus instructor-paced groups in person- 
alized systems of instruction. For example. Reiser and Sullivan n?77) 
examined the performince of student and instructor-paSed grouprof u^^^^ 
graduate political science students on unit quizzes and fina '^lxam 
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scores. In the self-paced group, students could take unit quizzes when- 
ever they chose, while in the instructor-paced groups, students took the 
quizzes on target dates set by the instructor. Results of interest 
included (a) significantly more self spaced than instructor-paced 
students withdrew from the course, (b) instructor-paced students tended 
to score higher on the final exam than self-paced students, but quiz 
scores did not differ between the groups, and (c) instructor-paced 
students tended to have more favorable attitudes toward the course than 
self-paced studeniS. Reiser and Sullivan attribute these results to the 
fact that self-paced students lacked the necessary skills to pace them- . 
selves consistently. Thus, these results support the need for time 
management skill training in self-paced environments and for some form 
of targeting system for helping students achieve a steady and consistent. 
' pace. 

Consistent with the latter suggestion, Pascarella (1977) compared 
course achievement and attitudes of university calculus students who 
were randomly assigned to a personalized system instruction (PSI) versus 
convehtional classroom groups. In the PSI group, students were given 
a suggested schedule for completing unit tests to help reduce procras- 
tination, whereas conventional stiiHents had a formal unit test schedule. 
In contrast to the Reiser and Sullivan (1977) study, Pascarella found 
that PSI students had higher scores than conventional students on both 
course achievement and attitude measures. These findings support the 
importance of providing a targeted completion schedule for reducing 
student procrastination and improving performance in a self-paced 
environment. 

Two additional studies which compared student versus instructor- 
paced groups in personalized instructional systems were conducted by 
Fernald (197b) and Bijou, Morris, and Parsons (1976). In the Fernald 
study, introductory psychology course undergraduates were assigned to one 
of three PSI f'»ature combinations: (a) teacher or student pacing, (b) 
perfection or no perfection requirement, and (c) much or little social 
contact with an undergraduate teaching assistant (TA). Results indicated 
that student-paced groups scored higher on weekly quizzes but not on mid- 
term exams than teacher-paced groups, students preferred TA contact and 
student pacing over no TA contact and teacher pacing, and the perfection 
requirement did not significantly affecf student performance. Contrary 
to these findings. Bijou et al . set up a weekly point system for 
rewarding students who stayed op schedule in an undergraduate PSI child 
development course, and found no performance differences for self -paced 
versus instructor-paced groups. Bijou et al. also found that the point 
system students paced themselves evenly through the course materials, 
while the no-point system students procrastinated. These findings 
suggest that external reward systems may be as effective as a targeting 
system for reducing student procrastination in self-paced system:. 

In the area of studies which have examined various self-monitoring 
strategies in self-paced PSI environments. Croft. Johnson, Berger, and 
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Zlotlow (1976) investigated the effectiveness of (a) weekly monitoring 
of course progress by instructor, (b) bi-weekly monitoring of course 
progress by instructor, (c) self-monitoring by students, and (d) no 
monitbring. In a comparison of the number of testing sessions required 
to complete the course and total points earned. Croft et al. found that 
students in all monitor:ng conditions performed better than students in 
the no-monitoring condition on both performance measures; however, no 
discernible differences were established between the monitoring 
conditions. ^ 

, A similar study on the effects of various self -monitoring conditions 
in an undergraduate PS I course was conducted by Yates and Zimbardo (1977) 
Irt this study, students were assigned to one of four conditions: (a) a 
self-monitoring group, who charted daily and cumulative time spent study- 
ing course-related materials, (b) a surveillance self-monitoring group, 
who additionally had to present their charts weekly to teaching assist- 
ants, (c) a group that received incentive points in addition to self- 
momtoring and surveillance by a teaching assistant, and (d) a control 
group who did not monitor their studying and were not informed of experi- 
mental mai.ipulations. The results of this study indicated that only 
students in conditions (b) and (c) performed better than control students 
on unit tests, suggesting that some type of surveillance of time manage- 
ment skills is required. 

Myers (Note 8) reviewed a number of studies indicating that 
students' learning and study behaviors can be significantly modified by 
self-monitoring techniques, such as having students observe, record, 
and/or graph their study activities. On the basis of this review, he 
investigated the relative effectiveness of (a) self-monitoring, (b) a 
combination of self-monitoring, self- reinforcement, and self-punishment, 
and (c) no monitoring on the midterm examination performance of male 
and female college students in chemistry and calculus classes. Relevant 
findings were that self-monitoring alone was as effective or moreso 
than the combination of self-monitoring, self-reinforcein6nt, and self- 
punishment in both improving students' test performanfcfe and increasing 
their reported study times. These findings suggest that the use of self- 
monitoring strategies such as charts or graphs should help maintain 
desirable time management skills throughout a self-paced course. 

dk In addition to studies on strategies for assisting stu 'ents to 
become more self-regulated and responsible for their learning, the 
effects of giving students data on completion times in a seliF-paced 
system are relevant. Johnson, Salop, and Harding (1972) found that Navy 
students given predicted times and incentives based on whether they 
completed lessons in less time than was predicted completed the course 
in 17 percent less time with no differences in final performance, as 
compared with control students. There were, however, attitude 
differences between the two groups which suggest that quite different 
motivational factors may have been operating. In a similar vein, Colton 
(1971) compared tiniR and achievement scores of college students who 
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were or were not given information about hpw long it took other students 
to finish 22 self-paced, criterion referenced tasks. Students given 
time information completed six tasks in significantly less time, but 
performed significantly less well on the criterion tests than did 
students not receiving the completion time information. These results 
suggest that time management skill training is more effective in the 
context of an instructional environment in which targeted completion 
times and rate of progress information are balanced with the importance 
of maintaining required achievement levels. 

Two related studies investigated the effects of goal-setting in- 
structions on student achievement in conventional courses. Gaa (Note 9) 
gave one group of tenth grade English class students weekly individual 
goal-setting conferences, during which they set goals for the next week s 
activities and received fu-edback on their performance and progress in 
attaining previous goals. A control group did not have the conferences, 
but received the same in-class instruction. Students in the goal -setting 
group had higher criterion test scores and attitudes toward the course 
than control group students. In discussing the attitude data, Gaa inter- 
preted the results to mean that the goal -setting group had higher 
motivation. In addition, goal-setting students increased their feelings 
of internal locus of control as compared with control students. Freeman 
and Niemeyer (Note 10), on the other hand, found no significant 
differences in criterion test scores as a result of goal -setting instruc- 
tions. 

Anderson (1976) investigated the differential effectiveness of 
mastery and non-mastery learning strategies in altering students time- 
on-task-to-criterion. When provided extra help (student tutors) in 
early course units and a mastery learning strategy (85 percent correct 
on within-unit tests) with built-in review proced^^es, students with 
lower entering skills attained equality with higher ability students on 
both attainment levels and amount of on-task time-to-critenon by the end 
of the third unit. In addition, students in the mastery learning 
condition spent less time on task following this early tutoria help than 
did students in the non-mastery condition. These findings imply that 
early task-relevant skill t-aining "costs" are amortized quickly when 
students begin using these skills to improve their performance. This 
suggests that early training in time monitoring and management can, on 
its own merits, reduce total tra.ning time by increasing the time 
students spend on task-relevant versus task-irrelevant activities. Comb- 
ining this skill training with completion time predictions and progress 
feedback (aS described by Johnson et al., 1972) could then be expected 
to result in even greater time reductions. 

1 2 3 Study Skills Training . Poore and Pappas (Note 11) point out 
thdt ineffective study is one of the most serious and persistent problems 
students have, regardless of ability level. It is reasonable to expect 
that study problems would be compounded in the novel envi .'onrnent of IMI. 
Support for the prevalence of study skills problems is provided by 
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several recent studies which have attempted to modify students' study 
habits and skills by a variety of training approaches. For example, 
Briggs, Tosi, and Morley (1971) examined the effects on subsequent grade 
point average (GPA) of exposing twenty freshmen women classified as 
having a deficiency in entrance exam scores, high school grades, or class 
rank to either an experimental study program or a no-treatment control 
condition. Over a 5-week period, experimental students were first 
exposed to procedures in which they incrementally increased their study 
time in each of five courses to 1 hour per day. After coming up to 1 
hour of study time per day per course, the Survey-Question-Read-Recite- 
Keview (SQ3R) study method was introduced to help the students use their 
study time more efficiently. At the end, of the semester, experimental 
students had significantly higher GPAs than control students in the five 
academic areas. 

McReynolds and Church (1973) investigated the relative effectiveness 
of regular counseling, experimental self-control counseling, experimental 
study skills development, and a no-treatment control condition for 
improving study skills scores, grade point average, reading scores, and 
Rotter I-E scale scores for" college underachiever volunteers. The 
regular counseling group received general academic counseling; the self- 
control group received training in the rationale and use of self- 
contracts for managing and reinforcing study goals and behaviors, as well 
as training in the general study techniques of the Robinson SQ3R study 
method; the study skills group received in-depth training in the 
Robinson SQ3R study method for a total of 10 treatment sessions. Results 
of interest were that (a) students in the two study counseling groups had 
improved study skills scores as compared to the regular counseling and 
control groups, (b) students in the two study counseling groups had im- 
proved GPAs as compared to the regular counselling and control groups, 
with more improvement for the study skills group than for the self- 
control group, and (c) no differences were found between the groups on 
I-E scale scores or reading test scores. 

A similar study by Groveman, Richards, and Caple (1977) compared 
the relative effectiveness of six conditions for improving GPAs: (a) 
no treatment control, (b) attention-placebo control, (c) study-skills 
counseling, (d) lengthened study skills counseling, (e) behavioral self- 
control, and (f) behavioral self-control plus study skills counseling 
groups. The study skills counseling consisted of group sessions in 
which students were taught skills in textbook reading, study scheduling, 
note taking, test taking, and writing. The self-control training 
emphasized self-monitoring, progressive relaxation, stimulus control, 
self-instruction, and self-reinforcement skills training. Experimental 
sessions met for 2 hours per week over a 4-week period or for 4 hours * 
per week over a 4-week period for groups (d) and (f). Due to method- 
ological problems of having only one therapist for all experimental 
conditions, the results of this study were suggestive rather than 
definitive, with marginal significance levels. The only finding 
reaching significance was that students in the behavioral self-control 
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treatments had improved GPAs as compared to the other groups. These 
findings do suggest, however, the potential viability of study skills 
training and behavioral self-control training for improving student self- 
management skills. 

Two additional studies suggestive of the effects of study skills 
training were conducted by Larkin and Reif (1976) and Gadzella, Goldston, 
and Zimmerman (1977). In the Larkin and Reif study, college physics 
students were taught general skills for studying scientific text, in- 
cluding how to state the characteristics of relations, give examples, 
interpret relations, make discriminations, and use equivalent forms of 
the relation to find or compare values. This training was accomplished 
by textual materials which introduced new relations, asked questions 
about the relation which required active student demonstration of the 
preceding abilities, and provided correct answer and explanatory feed- 
back. A control group learned the course materials as they normally 
would. Major findings were that students with skill training (a) 
significantly improved students' ability to acquire understanding of 
new relationships as compared to control students, (b) were able to 
effectively apply their new skills to the remaining physics textual 
materials and were less dependent on extensive instruction, and (c) 
retained their new -skills, as assessed by a 2-week followup measure. 
Implications of thii study are that although the initial skill training, 
required some studerit effort, this effort paid off in increased student 
efficiency in subsequent instruction. In addition, the authors point 
out tt.ot since studelts do not acquire these types of learning skills 
automatically, direct training of re' .van t skills is necessary. 

In the Gadzella et al. (1977) . .u^, the effectiveness of giving 
academically successful collene students study technique guides and 
quizzes, with respect to changing their perception of their study skills 
or their semester grade point average, was investigated relative to 
matched control students. Results of interest were that although 
students in the two groups did not differ in semester grade point 
average, students' perceptions of their study skills become more positive 
for the study skilU traininn group. The authors suggest that this 
increased confidence in their study skills and insights which the 
students received on how they could improve their study skills are well 
worth the additional time required for study skills training. Thus, 
even though no perfonnance gains were found, there would appear to be 
an advantage in improved student attitudes and confidence regarding their 
skills as a long-term effect of study skills training. 

The work of Dansereau and his associates suggests that there are 
many components of good study skills behavior that should be taken into 
account in attempts to improve student performance through specialized 
training in skills relevant to CMI environment requirements (Dansereau, 
Actkinson, Long, Ji McDonald, 1974; Dansereau, Long, McDonald, i 
Actkinson, 1975; Dansereau, Long, McDonald, Actkinson, Ellis, Collins, 
Williams, & Evans, 197b; Dansereau, Long, McDonald, Actkinson, Collins, 
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Evans, Ellis, & Williams, 1975; Dansereau, Collins, McDonald. Diekhoff, 
GarlaM, Holley, Evans, Irons, & Long, 1977; Dansereau, Note'l2). For 
example, Dansereau's learning strategy materials covsr such topics as 
creating a good mood for learning, self-coaching skills, concentration 
management techniques, general study skills and strategies, and test- 
taking Skills. In a recent discussion of the results of this type of 
Skill training, Dansereau (Note 12) reports that such training can have 
a dramatic effect on improving j,tudent skills, but the need still exists 
to demonstrate the effectiveness of such training in terms of course 
performance gains. 

Weinstein (Note 13, Note 14, Note 15) has stressed the importance 
Of Improving student skills for remembering and processing information, 
as a study skill area directly related to learning outcomes. She has 
developed materials and procedures for teaching students the use of 
various mnemonic devices, sentence and imaginal elaboration techniques, 
analogies, paraphrasing, and techniques for drawing implications and 
creating relationships. The key concept in these materials is that of 
having the learner become actively involved with the information to be 
learned, such thatthe material becomes personally meaningful. Addit- 
ionally, the skill training materials are designed to be generalizabl? 
to a wide range of subject matter areas. 

The training materials developed by Dansereau and Weinstein capi- 
talize on recent advances in learning theory and a growing body of 
empirical research derived from the new focds of cognitive psychology 
and information processing approaches on the active role of the learner 
in the learning process. A major proponent of applying this frainework 
to learning is Wittrock (1974) and Wittrock and Lumsdaine (1977), who 
has advanced a generative model of learning. A basic premise of this 
model is that learners tend to generate their own meanings and per- 
ceptions about materials to be learned and that these meanings and 
perceptions will be consistent with their prior learnlmj. This 
generative model of learning is consistent with cognitive theory and 
has major implications for the design of skill training materials 
addressed at improving students' study skills. 

A recent study by Doctorow, Wittrock, and Marks (1978) demonstrates 
the success of active information processing strategies. It was pre- 
dicted that reading comprehension would occur when students were required 
to actively generate their own sentences about story paragraphs. As 
predicted, the results of two experiments indicated that the combination 
of inserted paragraph headings and instructions to generate sentences 
approximately doubled reading comprehension and recall as compared to 
a control condition. Frase (1977) and Johnson (1974) have also stressed 
the importance of active information processing approaches for enhancing 
student learning. 



Additional evidence for the effectiveness of learning strategies 
which actively involve students in the material or assist students in 



making the Information to be learned more meaningful to their existing 
cognitive structures Is provided by an Increasing number of empirical 
studies. These studies embody a wide variety of active learning 
strategies. Including (a) visual or verbal elaboration strategies (e.g., 
Andre & Sola, 1976; Delaney, 1978; Hunter, 1977; Lyon, 197: . Rasco, 
Tennyson, & Boutwell, 1975; Snowman & Cunningham, 1975), (b) generative 
underlining/or sentence generation strategies (e.g., Bobrow & Bower, 
1969; Frase & Schwartz, 1975; Rickards & August, 1975), (c) serial 
rehearsal training strategies (e.g., McCauley, Kellas, Ougas, & DeVellU, 
1976), (d) meaningful semantic processing strategies (e.g., Anderson, 
Goldberg, & Hidde, 1971; Bellezza, Cheesman, & Reddy, 1977; Mlstler- 
Lachman, 1974; Postman & Kruesi, 1977), (e) note taking or reorganization 
strategies (e.g., Arkes, Schumacher, & Gardner, 1976; Carter & VanMatre, 
1975; Fisher & Harris, 1973; Shlnmerllk & Nolan, 1976), (f) problem- 
solving strategies (e.g., Olshavsky, 1977), and (g) self-reference 
strategies (e.g., Melchenbaum, 1975; Rogers, 1977). Good literature 
reviews In this area are provided by Bernstein (1973), Faw and Waller 
(1976), and Shimmerllk (1978). 

A final area of concern In study skills training. Involves the 
learners' test-taking skills. This area Is particularly relevant to the 
requirements of a CMI environment, where the number of tests students 
must take are substantially more than required In conventional class- 
room environments. The development of a Test-Wlseness program to Improve 
students' general test- taking skills Is described by Woodley (Note 16). 
A combination of text and workbook materials provide students with 
general suggestions and Individualized practice In applying specific 
test-taking strategies. Individualization is provided by means of 
specialized diagnostic and branching routines within the text and work- 
book. In a preliminary pre-posttest evaluation of this Test-Wlseness 
program, Woodley (Note 16) found that students improved their test-taking 
skills as measured by a test-wlseness scale. Gross (1977) reports the 
results of a similar study in which eighth grade students who were 
taught three test-wlseness behaviors (risk taking, deductive reasoning, 
time using) improved both their test-wlseness behaviors and standardized 
achievement test scores, as compared to students who had not learned 
these skills. 

1.2.4 Instructor Orientation and Training. A major determinant 
of student success in acquiring skills necessary In a self-regulated, 
self-paced environment such as CMI, Is the quality of the student- 
instructor tutorial relationship. This quality, in turn, Is very much 
dependent on the instructor's understanding of, attitude toward, and 
skills In the new instructor roles required by self-paced and CMI 
systems. Although the literature reviewed In this area contains a 
number of suggestions for defining the role of the instructor In a CMI 
environment, as well as some suggestions for areas to be covered In 
specialized CMI instructor training programs, the description of exist- 
ing programs and/or data pertaining to their effectiveness are con- 
spicuously absent. Thus, the literature selectively reviewed In this 
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section will focus on those articles or reports which describe instruc- 
tor roles and skills required in CMI, and on suggested training pro- 
cedures. 0 

In support of the critical ity of the instructor's attitude and role 
in the successful transition of students to a CMI environment, McMillan 
(1977) discusses the importance of instructor feedback to the formation 
of positive student attitudes toward the instructional environment. In 
his study, male and female undergraduates were given high and low effort 
assignments within praise and no-praise instructor feedback conditions. 
Of primary interest were the effects of these conditions on the forma- 
tion of positive attitudes toward the learning materials, classroom 
climate, degree of effort in performing the 'J'i^ianments, instructor 
knowledge and attitude, and content of the learning assignments. The 
results indicated that the attitudes of students in the no-effort con- 
dition were not differentially affected by the feedback versus no praise 
feedback conditions, whereas praise was found to significantly improve 
the attitudes of the high effort condition students. The major implica- 
tion of these findings is that with the new demands and performance 
requirements inherent in a self-paced, CMI environment (high effort 
condition), the instructor's role as supporter and facilitator of 
positive student feelings toward the system becomes critical. In 
addition, Wittrock and Lumsdaine (1977) point out that in an instruc- 
tional environment in which the learner is responsible for his own 
learning, the instructor's role must become one of being responsible for 
changing the learner's inappropriate attribution of success or failure, 
and helping jthe learner attend selectively to the information to be 
learned and Construct meaning from it. That is, the instructor should 
facilitate student acquisition of active information processing 
strategies jind skills. 

A numbjbr of researchers have discussed the transition of instructor 
role requirements from those required in conventional classrooms to those 
required in computer-based environments. For example, Falzetta (1973) 
points out 'that, ideally, the computer frees the instructor to assist 
students iiji becoming more skilled and responsible human beings. The in- 
structor's! role becomes one of guiding the student to a love of learning 
and, as such, provides the humanization qualities in this new learning 
environmenlt. 

More /specific delineation of the range of roles required in a 
computer-based environment are also outlined by a number of other 
researchers in the area (e.g., Bunderson, Note 17; Dyer, 1972; Campbell, 
Note 18; Kooi & Geddes, 1970; Hansen & Harvey, 1970; Harvey, 1978; Lamos, 
1971; Stanchfield, Note 19; Wilson, Note 20). There is fairly general 
consensus that new instructor roles include (a) diagnostician of in- 
dividual student learning problems, (b) prescriber of special learning 
activities, (c) manager of instructional resources, events, or student 
progress, (d) learning psychologist, counselor, and advisor, (e) subject 
matter expert, and (f) author of instructional software. Opinions 




differ, however, as to the relative importance of each role and the 
percentage of time instructors need to spend performing the duties 
required by each role. For example, some researchers place emphasis 
on the learning diagnostician/counselor/advisor role (e.g., Campbell, 
Note la; Hansen & Harvey, 1970), while other researchers emphasize the 
importance of the instructional manager/designer role (e.g., Bunderson, 
Note 17; Dyer, 1972). 

Given the rather radical shifts in roles from conventional to 
computer-based environments, it can be expected that instructors will 
have doubts, fears, or negative attitudes toward these new roles— 
particularly if they lack the skills and knowledges required to 
adequately perform them. Hansen and Harvey (1970) detail the nature of 
these role changes and state that instructors will be doing more • 
strategizing, managing, individual counseling, discussing, specializing, 
and diagnosing functions and fewer correcting, lecturing, and disciplin- 
ing functions. The essential nature of these roles in facilitating the 
students' learning is stressed by Lamos (1971), along with the importance 
of instructors accepting the challenge of making these new roles at least 
as interesting and stimulating as their conventional roles. 

A recent study by Campbell (Note 18) is particularly relevant in 
that he analyzed new instructor roles within the Air Force Advanced 
Instructional System (AIS). In discussing the shift in instructor roles 
from lecturer, test giver, test corrector, giver of grades to that of a 
training guidance counselor and advisor of students, Campbell points out 
instructors will be faced with even greater challenges in their new AIS 
roles. 'Ae stresses the fact that instructors are not only key instruments 
in the successful implementation of instructional technology, but will 
continue to be the key in effectively causing learning. Thus, the 
importance of adequately preparing instructors for their new role require- 
ments is apparent. 

Although a number of researchers have stressed the importance of 
• adequately training instructors in their new roles in computer-based 
environments and the need for such training programs (e.g., Hansen & 
Harvey, 1970; Hursh, 1976; Stanchfield, Note 19; Wilson, Note 20), the 
literature contains few examples of the content and procedures to be 
used in such training, or the relative effectiveness of various training 
approaches. The seriousness his situation is reflected in Hursh's 
v197b) discussion of effective..^ . data for personalized systems of 
instruction, and his conclusion that this effectiveness is related to 
the adequacy of instructor training in the specific behaviors required. 

One example of a training program relevant to CMI instructor skills 
is provided by Cohen, Emrich, and deCharms (1977). Teachers vffire taught 
to differentiate among four possible teaching styles and encouraged to 
use those styles which fostered more positive personal causation skills 
(e.g., independence, responsibility for own learning) in students. 
Elements of teaching styles felt to foster personal causation included 
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a flexible approach to learning which stimulates problem solving be- 
haviors, exploration of learning alternatives, and active, cooperative 
' classroom activities. Results of this study indicated that whereas 
teachtrs Jemonstrated knowledge of the^ styles, they did not later trans- 
fer the behaviors learned in the workshops to the classroom. Cohen et" 
a1. argue this was because one cannot assume that if someons knows new 
skillb, tney also want to, and know how to, apply them. Thus, L is 
stressed that any successful teacher training program should cover three 
components— that it should move beyond merely cognitive understanding to 
the want to, know how to, and try to dimensions. 

Iji a similar study. Gall, Ward, Berliner, Cahen, Winne, Elashoff, 
and Stanton (1978) examined the effects of three specific teaching skills 
(redirection, probing, and higher order cognitive questioning) on student 
learning, using 336 sixth grade students. Their results suggest that 
training packages for "teachers are most effective when thoy provide 
generalizable skill training, rather than, training in discrete skills 
req^ired in specific teaching situations. Two programs suggestive of 
the types of generalizable skills required In individualized instruc- 
tional situations are reported by Cadenhead (1976) and Rash and Grimm 
(1976). Cadenhead describes a pilot teacher training program in which 
teachers were taught four primary roles: diagnostician, facilitator, 
interactor, and innovator. Rash and Grimm, along with describing the 
implementation of a new self-directed learning program for junior high 
school students, describe a preservice teacher training program to pre- 
pare teachers for the changes they must make In their instructional 
practices. This training program included team planning activities in 
which teachers practiced acting as student learning advisors and 
practiced planning student learning activities, establishing student 
performance criteria, and planning strategies to increase the effective- 
ness of each student's learning program. These programs, then, support 
the feasibility of workshop formats for training teachers In new skill 
requirements, including the use of group process variables to stimulate 
instructor involvement and interest, and practice exercises to help insure 
the transfer of new skills to the instructional environment. 

1.2.5 Summary of Literature Re '* iew. This selective literature 
review focused on that literature suggestive of areas and approaches to 
be used in the development of student skill training materials and in- 
structor orientation and training procedures designed to facilitate 
student adaptation to a computer-based training environment. Some of 
the major implications derived from this review for the present project 
are^^ summarized as follows. 

T. A training package for orienting students to the novelties of 
a computer-based environment should present a realistic overview of 
system components, address common misconceptions abou** computer capa- 
bilities, describe the behaviors required in the new environment, stress 
t J self-directed and self -management skills required, and have a 
style which is warm and sensitive to student needs. Such a package 



would be expected to elicit positive student attitudes toward the 
system, which would hopefully be reflected In Improved student perfor- 
mance. 

2. The training of students In those skills required to manage 
tht time efficiently and effectively should address such areas as the 
learner's responsibility In a self-paced system, setting Mme and per- 
formance goals, establishing regular contacts with the Instructor to 
discuss progress, procedures for monitoring and charting rate of progress 
to help maintain time management skills throughout the course, as well 

as the specific skills Involved In efficiently completing cours? objec- 
tives. The literature also suggests that such training is most 
effective when combined with system or Instructor generated completion 
time targets. Such a time management package would be expected to 
result In Improved training efficiency. In the form of reduced course 
completion times, as compared with the absence of the package. 

3. Training In the specific student study skills required In a 
self-paced, computer-based Instructional system should emphasize the 
Importance of behavioral self-control techniques, as well as train 
stuilcmts In active Information processing approaches that will allow 
them to become actively and meaningfully Involved In the material to be 
learned. Specific study. ski 11 areas suggested by the literature as 

'^potentially relevant to a 'computer-based environment Include training In 
reading coniprehenslorr skills, memorization skills, concentration manage- 
ment skills, and test taking skills. Specific study skills training 
for students deficient In particular skill areas would be expected to 
Improve both student achievement and progress. 

4. Orientation and training of Instructors In those skills required 
to facilitate student learning In a self-paced, computer-based environ- 
ment should focus on those roles most directly Involved In the student- 
Instructor tutorial relationship. I.e., those skills required in their 
roles as diagnostician, tutor, and prescrlber of Instructional events. 
Addltlona'ily, Instructor orientation and training should include those 
procedures and strategies most conducive to the formation of positive 
instructor attitudes and the transfer of their new skills to the train- 
ing environment (e.g., workshop format to foster positive qroup pro- 
cesses, active participant Involvement and practice of sk1\!s in 
representative examples). This type of Instructor orientation and 
training procedure would be expected to facilitate student acquisition 

of skills required ir the CMI environmervt. 

1 . 3 rr oject Context: The Air Force Advanced Instructional System 

The context for the Computer-Managed Instruction Student Skills 
Project was the Air Force Advanced Instructional System located at 
Lowry Air Force Base, Colorado. The Advanced Instructional System, or 
AIS, is a prototype, multi-media, computer-based instructional system 
designed to Improve the effectiveness and efficiency of Air Force 
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technical training and to provide an operational research facility for 
assessing Innovations in Instructional technology. The system supports 
four technical training courses representative of the range of cognitive 
and performance skills required by enlisted Air Force personnel. An 
adaptive instructional decision model utilizes state-of-the-art computer' 
hardware and software, as well as currently available statistical 
methodologies and instructional procedures, to provide instructional 
management and individualized assignments to alternative instructional 
materials. 

1.3.1 AIS Course Structure . Each AIS course Is divided into 
"blocks" of instruction which may require anywhere from 1 to 15 days to 
complete. Each block contains a number of lessons and a comprehensive, 
end-of-block test. Within a block, lessons are arranged in a hierarchy 
based on their prerequisite relationships. A typical hierarchy 
resembles a set of parallel chains diverging and converging on certain 
pivotal lessons, and a student may alternately work on lessons in two 
or more parallel chains. 

The Dasic unit of Instruction is thu lesson. Each lesson consists 
of a set of objectives » two or more forms of a criterion test, and 
typically, a self-test for student evaluation of his or her understanding 
of the lesson before taking the criterion test. A lesson's instruction 
is provided by one or more modules, each of which teach the same 
lesson objectives and cover the same lesson content. Where two or more 
modules are present, they represent alternative instructional treatments 
or strategies. Depending on the lesson content, objectives, and nature 
of the treatment, a module may be a progratnned text, an elaborated 
technical order, or an audio-visual presentation. 

1.3.2 AIS Stujdent Scenario . A student's fmT'experience with the 
AIS is when he or she is administered a course-specific preassessment 
battery consisting of cognitive and affective measures considered to be 
predictive of students' performance in the course. Preassessment test 
forms are processed at an AIS management terminal and the student's 
initial course completion time and targeted days per block are printed 
out. 

Following preassessment, the student requests his or her first 
assignment by submitting a Forward-Going Assignment request at a manage- 
ment terminal. At this point, the student is enrolled in the course but 
has not yet entered a block containing actual course content. First, 
therefore, the System selects the block in which the student is to 
start work. Since the student has not yet completed any course work, 
only those blocks which have no prerequisites are selected. The student 
is then assigned to an appropriate learning center and home carrel. 
Finally, the student is assigned a specific lesson and an alternate form 
of that lesson's criterion test. 

Lesson assignments are determined by two major components of the 
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System— the Adapter and the Resource Allocation Model. The Adapter 
attempts to select, for each student and for each assignable lesson, the 
one module or instructional treatment which Is most appropriate for 
that student. This decision can be based on a variety of rules, e.g., 
select the module which the strdent is predicted to complete in the 
shortest time given a predicted passing score on the criterion test. 
Each alternative module is given a weight indicating its relative pre- 
ference. Alternatively, the Resource Allocation Model assigns prefer- 
ence weights to modules for which the required resources are available, 
in order to minimize the assignment's impact on the availability of in- 
structional resources. Final lesson and module selection is based on a 
compromise between the tv(o sets^of preference weights. 

Having received the first printout (Student Status Report), the 
student reports to an instructor in the learning center assigned, 
obtains the instructional resources required for the assigned lesson and 
module, and begins work at th;^ assigned home carrel. After studying the 
lesson materials, the student completes a multiple-choice lesson self- 
test and reviews the material pertaining to questions answered incorrect- 
ly. The student completes the version of tne lesson criterion test 
assigned and submits the test form to a management terminal. The 
resulting Student Status Report details the student's performance on 
the criterion test (percentage total score, item; missed, objectives 
failed, and the pass/fail decision) and gives the next assignment. If 
the test criterion was not met, the student is reassigned the same 
lesson and an alternative version of the criterion test. Otherwise, 
the lesson, module, and test selection procedures are repeated and a new 
lesson assignment is given. 

When the student has completed all lefsons in the block, a Block 
Review lesson is assigned. When the student indicates readiness f6r 
the block test, one of the alternate forms of this test is randomly 
assigned. While lesson tests can be vieweo as primarily diagnostic 
tools, end-of-block tests serve a certification function. If the student 
does not meet the block test criterion, he or she is reassigned to the 
block in a status whereby assignments are made by the instructor rather 
than by the System. If the block decision is "go," the block selection 
logic is repeated and the next block of study is assigned. 

the student's continued progress through the course is essentially 
a repetition of the preceding events. Two exceptions are that question- 
naires designed to assess students' attitudes w/ard the AIS are 
assigned following the first and last blocks in the course. 

1.3.3 AIS Student Progress Management Component . During the 
design and development phases of this project (early to mid 1977), soft- 
ware to support a Student Progress Management Component (SPMC) had been 
designed and was being developed. Prior to that time, the AIS supported 
a simple form of student progress monitoring and reporting which was 
recognized as being inadequate, one major concern being that it was not 
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individualized. In addition, three specific problems shaped the 
characteristics of the SPMC. First, the general opinion was that AIS . 
students could, if motivated, complete more course work than they were 
doing during the regular, 6-hour shifts. A related concern was to in- 
crease the amount of work students completed off-shift. Lessons were 
sometimes voluntarily completed out of class, and students who failed 
one or more block tests were assigned off -shift remedial training. If, 
however, a student passed the block tests and progressed at a 
"reasonable" pace, there was no pressure to continue work off-shift. Tuc 
third problem concerned prediction of course completion dates. Students' 
anticipated course completion dates were needed 10 days in advance fo»* 
"out-processing," but variability in the time to complete was so great 
that useful predictions required unattainably high correlation coeffi- 
cients. 

The SPMC was intended to address each of the foregoing problems, as 
well as the students' expressed desire for some method for measuring 
their progress which was uniform throughout the course (see Sections 
2.1.1.1 and 2.1.2.1 of this report). The software for the SPMC was to 
generate a target course completion time for each student which was pre- 
dicted on the basis of pre-course ability, attitude, interest, and ba^ck- 
ground information; and which assumed that some lessons would be 
completed as homework. Daily feedback was to be provided to *;ie student 
and to the student's learning center instructor, and remedial training was 
to be assigned on the basis of poor rates of progress relative to each 
student's target or following block test failures. Course managers were 
to employ available positive and negative incentives to manage students 
to on-target course completion. v 

As the SPMC was implemented within the AIS, a student's course com- 
pletion time is predicted by a multiple linear regression equation em- 
ploying data from each course's preassessment battery. A "Policy 
Function" converts this predicted time to a target time. If, for 
example, course management has determined that course completion times 
can and should be shortened by 5 percent, students' target times are set 
to 95 percent of their predicted course completion times. After a 
number of additional steps which allow for various paths through the 
course and changes in course content, the end result is an individualized 
target rate, in the form of a standard score, corresponding to the 
student's target course completion time. 

Time spent in the course Is updated when the student completes an 
assignment and the target time for the amount of work completed is up- 
dated when progress feedback is requested. The difference between these 
two times is the extent to which the student has deviated from his or her 
target rate. 

The initial SPMC printout occurs when the target rate is first com- 
puted, following submission of the student's last preassessment test 
form. Target times for each block and the student's total course com- 
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pletion time are listed in units of days and tenths of days. This print- 
out is delivered to the student's learning center instructor and the 
information on this printout is used by the student in completing the 
progress monitoring procedures described in the Time Management Module 
(see Section 2.0 of this report). 

The student's first Status Report of each day contains the days and 
tenths of days of the course completed, and the days and tenths of days 
spent in class. The amount of work constituting a "day" of the course 
is, of course, a function of the student's target rate. Each student s 
rate of progress is also reported on the learning center roster which 
instructors receive at the beginning of each shift. Roster information 
includes the number of days and tenths of days remaining to the 
student's targeted course completion date, and the number of days and 
tenths of days by which the student is ahead of target. If "days ahead 
of target" is negative, the student is behind target and instructors 
can easily detect students who are falling behind in their course work. 

If an instructor decides that a student's target rate should be 
reset, a target change request can be forwarded to the course Database 
Manager, who changes student targets through an interactive editor. 
Although it would have been feasible to alter students' target rates 
automatically on the basis of their actual rates of progress, specific 
intervention by the instructor was purposefully required. Given the 
variety of reasons why students may be behind or ahead of their target 
rates, it was reasoned that the instructor is in a better positipn to 
determine the correct action than .is the SPMC software. 

2.0 ORIENTATION TO CMI ANU TIME MANAGEMENT SKILL TRAINING 
2.1 Problem Definition 

To determine the characteristic problems students experience in a 
CMI environment and further define training requirements for an Orien- 
tation to CMI Module and Time Management Module, a series of student 
interviews were conducted in the first phase of the project. Students 
selected for these interviews were from ont of four groups from the 
Inventory Management (IM) or Materiel Facilities (MF) courses: (a) IM- 
Good, (b) IM-Poor, (c) MF-Good, (d) MF-Poor. "Good" students were those 
completing the course faster than the average rate with above average 
grades, and "poor" students were those completing the course at a slow- 
er than average rate with below average grades. All selected students 
were working in the last half of the course and were selected on the 
basis of AIS records. 

The interviews were sem -structured discussions wherein the investi- 
gator asked a set of predetermined open-ended stions on a one-to-one 
basis. Twenty IM students and six MF studenti .-re interviewed for 
approximately 30 ninutes each. During the int. /lew, dif-^ussion centered 
around the AIS in 'jeneral and the student's specific problems in the 
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course. The results of student responses to the 18 questions are 
summarized in Table 1. 

Conclusions drawn from these interviews were as follows: (a) 
students need a method of measuring their progress which is uniform 
throughout the course, (b) in conjunction with progress measurement, 
students need to be taught some basic time management skills, (c) 
students need to be given information on what to expect in an irdividual- 
ized, computer-managed course, and (d) students need to be told th=»t 
trying for the best possible grades, without appropriate concern for 
learning time, is not necessarily the way "to do better" in a CMI course. 

' 2.2 Module Design and Development 

A complete description of the concepts and procedures used in the 
design and development of the Orientation to CMI Module and the Time 
Management Module can be found in AFHRL Technical Report No. AFHRL-TR- 
79-14, along with a copy of the instructional materials and evaluation 
instruments. Thus, the present report will provide only brief descrip- 
tions of the objectives of each of these modules, the instructional 
strategies used and the module formats. 

The Orientation to CMI Module was designed to be assigned to 
students at the beginning of their training course and to serve as the 
first part of an Orientation Lesson. The second portion of this lesson 
was to be the Time Management Module. Given that these modules under- 
went separate formative and summative evaluations, they will be described 
separately. 

2.2.1 Orientation to CMI Module . The major design goals for this ' 
module were to (a) improve students' attitudes toward CMI and (b) reduce' 
course completion time. It was hypothesized that if students were given 
a solid understanding of novel CMI features and instructional procedures, 
they would have a better appreciation for the benefits of this instruc- 
tional system and would waste less time trying to figure the system out 
for themselves. It was also felt that students would' begin using the * 
CMI-provided instructional tools earlier in the course and would, accord- 
ingly, spend more time learning the instructional materials. A sub- 
ordinate goal for this module was to provide students with appropriate 
role models for increasing acceptance of their responsibility for learn- 
ing in a CMI environment. 

2.2.1.1 Module Objectives . The '.tated objectives, listed in the 
front of the module were as follows: (a) You (the student) will be able 
to recognize the major differences between a computer-managed instruc- 
tional system and a group-paced (conventionil ) instructional system; (b) 
You will be able to recognize those benefits and features of a computer- 
managed instructional system f-hat help you to be responsible for what 
you learn; (c) You will be atie to recoqnize how an efficient student is 
different from an inefficient student in a computer-managed instructional 
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TABLE 1 

Student Opinions of a CMI Environment 



QUESTIONS 



STUDENT Rt:S(>ONSl:^ 



1. What do you think about 
AIS? 

2. How do you like the idea 
cf self-pacing? 

3. Dj you have a feeling for 
now you are progressing? 



5. 



Do you think anyone 
cares about how you are 
doing? 

What strategies do you 
use to plan your time? 
Is it easy or difficult 
to plan time? 



6. Do students care about 
doing well? 



If you wanted to do 
better, what v^ould you 
do? * 

What is the biggest 
problem students have 
with this course? 



All students interviewed liked AIS. 



IM-G, MF-G & MF-P liked self-pacing. ^ 
IM-P students preferred group-pacing. 

ftost students had devised some method 
of measuring their progress such as ^ 
competing with peers, counting number 
of lessons done compared with those 
remaining, and/or calculating how much 
they had learned. 

I'lost students believed their instruc- 
tors cared about how they were doing, 

IM-G students ge.. uily used some time 
management strategies. About half of 
the IM-P students planned their time. 
fJone of the MF-G or MF-P students 
planned their time. 

A majority of all students believed 
that other students wanted to do well 
in the course. 

A majority of all students indicated 
that they would "take more time." 



"Motivation" was mentioned by all 
students, although other problems such 
as reading comprehension and technical 
difficulty were also mentioned. 



^ IM-G = IM Course Good. 
IM-P = IM Course Poor. 
MF-G = MF Course Go»<i. 
MF-P = MF Course Poor/N.^ 
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TABLE 1 (Cont'd.) 



QUESTIONS 



9. 



10. 



11. 



12. 



13. 



14, 



15, 



If you were advising a 
student just starting 
this course, what would 
you tell him/her to do 
or not do? 



If you were in charge of 
this course, what changes 
would you make? 



How do you fet about 
having to reac . per- 
formance objectives? 

Was there a point in 
the course where you 
felt unmotivated or lost 
interest? 

How do you feel about 
the computer grading 
your tests and giving 
you the next assign- 
ment? 

How do you feel about 
your instructors? 



Tell me about your 
study habits? 



The majority of all students in- 
dicated they would suggest that new 
students "ask questions." "Good" 
students suggested that new students 
not listen to the rumors about how 
easy/difficult specific sections of 
the course were. "Poor" students 
suggested that new students take 
notes. 

Students from each sub-group 
mentioned giving students an option 
of group-paced or self-paced in- 
struction. Poor students expressed 
a desire for more introductory 
information. 

A majority of all students indicated 
that the standards were acceptable. 



IM-G, IM-P all mentioned Block III. 



A majority of students liked the 
computer grading their tests and 
giving them the next assignment. 



All students interviewed had very 
good conments about their 
instructors. 

Students in all sub-groups generally 
answered the embedded questions and 
took notes. Many IM-G, IM-P & MF-P 
students used the objectives to 
review for a test whereas none of the 
MF-G students, used this method. IM-G 
IM-P & MF-P students, more often 
used the block review lesson than did 
MF-G students. 
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TABLE 1 (Cont'd.) 



TTDDcnrmpoNSEs 



16. 



17. 



How do you use the 
objectives? 



What motivates you to 
complete the course? 



ly. 



How would you feel 
about having a target 
date set for you? 



A majority of all students read the 
objectives and many of them used the 
objectives to review for a test. 

IM-P and IM-G students were most often 
motivated by pride. MF-G and HF-P 
students were most often motivated by 
external motivators such as "honor 
graduate" or "military career." Poor 
students in both courses were 
motivated by the realization that they 
were going home upon completion of the 
course. 

A majority of students the a 
target date would be a good 
motivational tool for students. 
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system; and (d) You will be able to identify good techniques for managing 
your time. 

2.2.1.2 Instructional Strategies and Procedures . The module was 
designed to be sufficiently general so that It could be used in a vanety^ 
of military CMI environments. It contained no references to the students 
specific course or training base and attempted to explain only those 
system features felt to be generic to a majority of CMI systems. It was 
intended that the module could be appended to the front of nearly any 
military CMI training course with few, if any, changes. 

The instructional technique used to convey a positive set for both 
the CMI environment and the new behaviors required in tins environment 
was a light, humorous, and persuasive style. Cartoon figures of male 
and female trainees were used in depicting self-talk sequences of 
efficient and inefficient CMI students. 

2.2.1.3 Module Format and Evaluation Instruments . The Orientation 
Module was entitled "How to be a Successful Student in a Computer- 
•Managed Instructional (CMI) System, O' Now You Are Responsible for What 

You Learn." The module was written in a narrative style with extensive 
use of cartoons and diagrams to avoid the appearance of a technical 
document and to provide for low density. Vocabulary and grammatical 
style was kept as simple as possible, and an 8 1/2" x 11" illustrated 
text format was used. 
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The module was divided into four sections, each corresponding to 
one of the four objectives. Section 1 compared a CMI system with a 
group-paced system; Section 2 described the skills required of students 
in a CMI system and the benefits of such a system; Section 3 described 
the typical behavior of efficient versus inefficient CMI students; and 
Section 4 presented a rationale for good time management techniques in 
a CMI environment. 

The evaluation instruments consisted of a ten-item pre/post 
Attitude Toward CMI Questionnaire and a bbst lesson test covering the „ 
objectives of both the Orientation and Trtme Management l^dules. The 
attitude questionnaire consisted of ten likert scale items focusing on 
the student's opinion of CMI. Item order was varied between the pre- 
and post-forms of the scale. A cne-page, readministf rable sunmary of the 
Orientation Module was also provided. This summary was designed to be 
posted in each learning center to remind students of appropriate be- 
haviors in a CMI environment. Copies of all these materials can be 
found in AFHRL Technical Report No. AFHRL-TR-79-14. 

2.2.2 Time Management Modul e. The major design goil for this 
module was reduction In course completion time beyond any reduction 
attributable to the AIS Student Proqress Management Component (SPMC). 
Subordinate goals were that (a) students would maintain their inriividuai 
progress tracking charts on a daily basis, (b) students and instructors 
would meet for scheduled Progress Counseling Sessions, and (c) students 
would express positive attitudes toward CMI in general and with the SPMC 
in particular. 

. 2.2.2.1 Module Objectives . The module listed three objectives: 
(a) You (the student) will be able to use the computer to help you see 
how you are proqressing toward your graduation day; (b) You will be able 
to keep track of your daily progress on a chart; and (c) You will be 
able to identify when to enter into a Progress Counseling Session with 
your instructor. i 

2.2.?. 2 Instructional Strategies and Procedures . A progress 
chorting technique constituted the primary strategy for promoting self- 
monitoring and self-ri.anagement student behaviors in a CMI environment. 
This progress charting technique centered around a graph called a "Course 
Completion Map." The axes of the graph corresponded to the rate-of- 
progress information listed on the students' daily Student Status 
Report. The abscissa was labeled "Days Spent in CUss," the ordinate 
was labeled "Days of Course Completed," and the divisions of the graph 
were in fifths of days. The student's Target Rate was shown as a 45- 
degree angle line extending froirt the origin out to the number of days 
targeted for course completion. The Map also contained a box wherein 
the student was to record the number of days he or she was targeted to 
spend in each block of the course (see Figure 1). The reverse side 
of the Course Completion Map contained instructions and a check list for 
Progress Counseling Session procedures (see Figure 2). Thus, this 
"Course Completion Map" was used in conjunction with progress feedback 
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FIGURE 1 
COURSE COMPLETION MAP 




FIGURE 2 

Progress Counseling Session Procedures 



1. Are you close to your 
Target Rate? 

2. If you are behind, do 
you see a way to get 
back on schedule 
fairly easily? 

3. If you are right on 
schedule, do you 
think you can improve 
your pace? 

4. If you are ahead of 
the Target Rate do 
you think you can 
continue at this 
pace and finish a 
little early? 

5. Given your answers to 
the above questions, 
what would be a 
reasonable but 
challenging goal? 
(Grad. date?) 



BLOCK 2 
CHECKPOINT 



YES 



GRAD. DATE 
CHECKPOINT 



NO 



YES 



NO 



NOW GO SEE YOUR INSTRUCTOR 
********************************************* 

PERFORMANCE CONTRACT RECORD 

Date \ I am days behind my Target Rate and agree to be 

back on schedule by . 

Date : I am days behind my T-arqet Rate and agree to 

be back on schedule by , 



Date : I am days behind my Target Rate and agree to 

be back on schedule by . 
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information provided by the SPMC to promote maintenance of students' time 
managoment skills. 

Four new learning center procedures were also created for use in 
conjunction with the Time Management Module. First, instructors were 
required to periodically check their learning center roster to determine 
which students were either 2 days ahead or behind schedule and then to 
initiate Progress Counseling Sessions with these individuals. The 
instructor was to assist fast students by helping them to establish a 
Goal Rate indicating their personal goal for completing the course. To 
assist slow students, the instructor could initiate a "Performance 
Contract/' an agreement between the student and instructor indicating 
that the student agreed to make up a specified number of days by a 
certain date. 

The instructor could also change student target rates felt to be 
inappropriate, as described in Section 1.3.3 of this report. In order to 
monitor the frequency and direction of the target changes, instructors 
were directed to contact the contractor to initiate target changes. 

The third new learning center procedure involved the documentation 
of Progress Counseling Sessions on a designated military form. This 
not only insured consistency when new instructors camo into the learning 
center, but it also provided a method for determining the extent to 
which the procedures were being implemented. 

The fourth procedure required instructors to periodically check 
students' Course Completion Maps to insure that they were accurately 
completed. 

2.2.2.3 Module Format and Evaluation Instruments . The Time Manage- 
ment Module was entitled "Time Management In a Computer-Managed, 
Individualized Course, or If You Don't Know Where You Are Going, How 
Will You Know When You Get There." Like the Orientation Module, the 
Time flanagement Module was written in a light narrative style with 
vocabui .ry and grammar kept as simple as possible. Explicit directions 
and examples were provided for the performance portions of the instruc- 
tion (i.e.. creating the Course Completion Map) and cartoon graphics 
and diagrams were employed extensively. The module was bound in an 
3 1/2" X 11" illustrated text format. 

Major concepts to be covered in the module narration were (a) the 
importance of completing the course quickly via utilization of efficient 
time nanaqement skills, (b) why and how each student is given a specific 
number of days to complete the course based on the student s potential, , 
(c) hoW^ students should manage their study time using the Progress 
Charting technique, and (d) how to schedule Progress Counseling Sessions. 

The evaluation instruments for this module included the previously 
mentioned post-lecson test and five open-ended attitude questions deal- 



mg with tjie student's opinion of the Time Managomant concept. Students 
were ?lso given a Time Sheet which listed the average tirne-to-complate 
each lesson of the course. A one-page summary of the Time Management 
.^ouule was printed on the reverse side of this handout. Copies of these 
instruments can be found in AFHRL Technical Report AFHRL-TrI.79-14. ' 

2.3 Fo'-mative Evaluation 

Fcrrnative evaluation activities for bdth the Orientation and Time 
Management Modules were conducted in two Phases: (a) a small group try- 
out in which the module was administered to students and instructors to 
determine whether or not the module accurately and efficiently conveyed 
.Che desired information; and (b) an operational tryout in which the 
module was placed ^n the appropriate position in the course hierarchy 
and administered to students who met the criteria for being assigned 
that particular module, in orJer to assess the effects of the module on - 
student perfornance and attitudes. Although appropriate rpvisions were 
made following each phase, the operational tryouts were essentially a 
form of sum-native evaluation. 

The procedures and results of the formative cvaluatiori for these 
two modules are extensively described in AFHRL Technical Report No. 
A(HRL-TR-7y-14. For that reason, the following sections will briefly 
sui.ii.iarize evaluation results and chanaes made to the modules based on 
tfiese results. 

c.3.1 Qriuntdtiori to CMI Module . In the interest of efficiency, ' 
snail (jroup tryouts for the Orientation Module and analysis of student 
study skill problems were conducted simultaneously via student and 
instructor interviews. A description of- these interview procedures and 
types of interviewees can be found in Section 3.1 of this report. In 
general, Zii students from four student groups in the IM and MF courses 
were mterviewed, along with 12 instructional staff members, and tv/o 
studc t eliminees. ' 

Pertinent small group tryout results included (1) mean time to com- 
plete the module was approximately 20 minutes, with a range of 16.1 to 
23. b innutes, (b) students expressed a need for embedded questions in the 
module, (c) students and instructors requested changes to several of the 
cartoons, and (d) a number of explanations of efficient student behaviors 
for CMI were in need of clarification. Based on these results, 
appropriate revisions to the Orientation flodule vvere made prior to 
operational tryouts. 

The opportunity for operational tryout of the Orientation to CMI 
Module materials was restricted to the Precision Measuring Equipment 
(PMt ) course since operational evaluations of the Study- Ski lis Modules 
were scheduled concurrently in tlie other three AIS courses. For 
approximately 6 weeks, data w re collected on block completion times 
and score-, and pre/post Attitude Toward CMI scores, using data from 



students who entered and completed the second CMI block in the course 
during this; period. To provide an evaluation of the relative effective- 
ness of an Orientation versus No-Ori.entation condition, students were 
randomly assigned to Module 01 or 02, respectively, for this first 
lesson of the block. 

£ 

. . At the completion of the evaluation period, data extracted for the 
criterion variables of interest revealed only nine cases for ttodule 01 
and seven cases for Module 02., When time constraints were applied to 
exclude unreasonable cases (exceiiive block completion 'wimes), the 
nurpber of cases for Module ►Ol dropped to four and the number of cases 
for Module. 02 dropped to six. Analyses on both the constrained and un- 
constrained data revealed no reliable differences between the groups on 
block times, block scores, or attitude measures. The results of the 
operational tryout, although somewhat 4nconclusive due to the small 
nuirSer of students available for this evaluation, are promising in light 
of anecdotal data supplied by both students and instructors. That is, 
comments concerning this module indicated that (a) there was a definite 
requirement for orienting students to the novel aspects of a CMI environ- 
ment .and (b) the module designed was positively received. Thus, the 
final characteristics of the Orientation to CMI Module are those 
described in AFHRL Tecnhical Report No. AFHRL-TR-79-14. 

2.3.2 Time Managem e nt Module . During small group tryouts of this 
module-k a total of 42 Ih students were administered the module on a one- 
to-one basis. In "addition to observing problems and soliciting comments 
while students were working on the module, the evaluator administered 
the criterion test and five-item attitude questionnaire. 

Results of relevance from the small group tryouts were that (a) 
although students received satisfactory criterion test scores and were 
uhle to initiate a Course Completion Map, they often had difficulty 
plotting their daily progress; (b) students expressed positive attitudes 
toward the module and that they thought it would benefit them; and (c) 
following the addition of two frames to the module which more thoroughlv 
covered the daily plotting of progress, the mean time for 27 of the 
ori'jinal 4'' students to complete the module was 57.1 minutes. Module 
perfonnance was considered satisfactory at this point, and ready for an 
operational tryout. 

Operational tryout of the Time Management Module was conducted in 
the IM course, during which time it was installed as the first lessen 
in the first block of a special formative evaluation version of the 
course. The module questionnaire was not administered since a standard 
Student Attitude Questionnaire was administered at the end of Block 1. 
During the operational tryout period, 64 students entered the evaluation 
version of the course and of these, 28 completed the first four blocks.. 
The number completing the remaining tuo blocks was too small for mean- 
ingful analysis. 
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Operational tryout results indicated that (a) first attempt module 
criterion test scores had a mean of 38.3. with only one student failing 
to meet the criterion of 60 percent, (b) first attempt lesson times were 
found to be unreliable due to the lesson's position as the first assign- 
ment in the course, against which administrative activities required at 
course entry were being charged, (c) students maintained their Maps with 
little difficulty and many indicated that they enjoyed keeping track of , 
their dail:y progress, (d) comparisons of completion times and block 
scores for the first four blocks with a comparable group of students 
entering the course during the same period yielded no significant 
differences, but were considered promising in that evaluation students 
completed in 3.75 hours less than control students, and (e) evaluation 
students had less favorable,' attitudes than control students on items 
related to time management, i.e.. they felt more pressure to complete 
the course quickly, felt less able to work at their own pace, and per- 
ceived their interactions with instructors to be less satisfactory than 
did control students. 

Interviews with 13 evaluation group students were also conducted as 
part of the operational tryout. From these interviews it was Iparned 
that many of the students did not understand the concept of completing 
the course as quickly as possible with minimal passing grades, and when 
it was explained to them, they expressed that they did not like this 
compromise. These results, along with other tryout results led to a 
number of revisions to the Time Management Module. Tliese included (a) 
a liKire indepth explanation of the differences between the goals of 
military technical training and public school education, (b) adding a 
handout listing average completion times for each lesson in the course 
to help students pace themselves on a lesson-by-lesson basis, and (c) 
including summarized directions for plotting daily progress, scheduling 
Progress Counssling Sessions, and defining a "Goal Line." The lesson 
test criterion was raised to 70 percent and a second parallel version 
was created. 

Z.A Summativo Evaluation 

For the purposes of this project, sunimative evaluation was con- 
sidered to be ,-\ second large-scale tryout in which data on approximately 
bO students per treatment were collected. Given time and resource con- 
straints, it was necessary to restrict full summative evaluation of 
only the Time Management Module portion of the total Orientation Lesson. 

'^•^•l Time Management Module . Summativo evaluation of the Time 
.Management flodule was conducted in two phases. During Phase I. the AIS 
was undergoing an extensive evaluation (Integrated System Test) and the 
utility of the Student Progress Management Component (SPMC) was a major 
consideration in this evaluation. The Time Management flodule had come 
to be considered an integral part of the SPMC and. thus. Phase I of the 
evaluation addressed the combined effects of SPMC and the Time Manage- 
ment flodule. Following completion of the Integrated System Test. Phase 
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JI of the evaluation addressed the effects of the Time Management Module 
per se in the presence of the SPMC. 

2.4.1.1 Phase I Evaluation Procedures and Results . During the 
Integrated System Test, students entering the IM course were randomly 
assigned to one of three versions of the course: a "Main Track" version 
in which all students were assigned the instructional mode originally 
developed for that lesson; an "individualized Instructional Assignment" 
version, in wriich students were assigned one of two to five alternate 
instructional modules on the basis of their predicted time and score 
under each alternative treatment; and a "Random" version in which 
students were randomly assigned one of the alternative modules. Since 
Main Track and Individualized Instructional Assignment were considered 
representative of two modes in which CMI might be used for military 
technical training, a question of interest was whether students would 
react differently to the, SPMC and the Time Management Module iir.der these 
two conditions. 

The SPMC and the Time Management Module were introduced halfway 
through the Integrated System Test. The "no progress management" period 
lasted a total of 57 class days. A con parable 57 class day period was 
dofined beginning with the implementation of the SPMC (and Tii.ie Manage- 
ment Module). Students who began the course and completed one or more 
blocks ill the Main Track or Individualized Instructional Assignment 
versions during the no-management and management periods constituted the 
control and evaluation groups, respectively. During the management 
period, the ilanagement Component Pol icy Function was set to 90 percent. 

2.4.1.1.1 Module Performance Data . The Time Management Module was 
moved to the second lesson in the first block to avoid contamination of 
the lesson tiding data. After about a month, however, course management 
moved the first two lessons out of the first block and administered them 
as part of the preassessment activities. Consequently, collection of 
module performance data was restricted to the first 2U cliss days of the 
evaluation period. 

Mean first-attempt lesson tine (based on tiie data of IOIj students) 
was o7.6 minutes witn a standard deviation of 21. G minutes. 

For 12b students wUh item level criterion test data, th - Alpha 
reliability of the test wcs found to be only .44U due primarily to six 
test items. Five of these six items had very low error rates and the 
sixth (pertaining to the frequency with which rate of progress data is 
provided) had a high (4b percent) error rate. These items were subse- 
quently revised and satisfactory reliability was found for both forms 
of the tost. 

First-, .tempt criterion test scores were available for 13b of t ? 
evaluation group students who completed the first block of tha cours . 
For those students, the mean f irst-att^»mpt score was Uine students 



(6.6 percent) failed to meet the criterion of 70 percent correct. 

2.4.1.1.2 Block Completion Time and Score Data . The question of ' 
primary interest concerned the hypothesized reduction in block and course 
completion times under Student Progress Management. Of secondary 
Interest was whether students would react differently to Progress 
Management ur.der Main Track versus Individualized Instructional Assign- 
ment. The block time data (block elapsed time less absence time), a 
measure of total course (cumulative times over six blocfks for sttijents 
having reliable data for all six blocks), and first attempt block test 
scores were, therefore, evaluated by means of two-by-two analyses of 
variance. The F tests of Interest in these analyses were the main 
effects for Management/Mo-Management and the interaction between 
Management/No-Management and assignment mode. ,Main effects for assign- 
ment were not of interest for this evaluation. 

none of the seven time comparisons or six score comparisons in- 
dicated a significant interaction between Management/No Management and 
assignment mode. All of the seven time comparisons and four of the six 
score comparisons did, hov/ever, indicate significant (p < .01.) main 
effects for Management/No Management. It was concluded that the effects * 
of Student Progress Management were not moderated by the two differing 
modes of assignment. Consequently, the data obtained under the two 
assignment modes were combined and only the main effect of Student Pro- 
gress Management will be discussed. The means and standard deviations 
of the block times and scores obtained under Progress Managemeri't' (Eval- 
uation Group) and prior to implementation of the Progress Management 
Component (Control Group) are presented in Table 2, along with the time 
anc' score differences between the two groups and the percentage time 
changes. 

As Table 2 indicates, the mean completion tine for Block 1 v/as 
20.00 hours for the evaluation group as contrasted with a mean of 23.75 
hours for the control group. This is a savings of 3.75 hours or 16.8 
percent (F(l,57lj) = 37.150, and p = .001). This is not, however, a 
clean comparison because of the changing content of the block: addition 
of the Time Management Module at the start of the evaluation period and 
removal of both the module and the first lesson three weeks later. 

The next two blocks, 2 and 3, both demonstrated significant time 
savings: 10.3 percent for Block 2 (F(l,505) « 15.760, p = .001); and 
13.7 percent for Block 3 (F(l,435) = K.145, p = .001). 

While still significant. Progress Management had relatively little 
effect on Block 4 times, resulting in only a 6.6 percent reduction 



An evaluation of the Individualized instructional Assignment (as con- 
trasted with Main Track) will be reported in McCombs et al . (Note 3). 
It should be noted that while only block level data are reported here, 
the questions of primary Interest in the McCombs et al. evaluation in- 
volved lesson level data. 
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(F(l,422) = 6.992, p = .003), While there is no obvious reason why 
Block 4 should be less susceptible to Progress Management, it may be 
that the "mid-course slump" was relatively impervious to the treatment. 

The final two blocks, 5 and 6, had a Pattern similar to the first 
three blocks: a 10. b percent reduction in Block 5 F .348 = 13.776, 
p = .001); and a 14.2 percent reduction in Block 6 (F(l,342) = i9.U19, 
P = .001). ^ 



For coiiiparing total course completion times, only data frorn eval- 
uation or control group students wiio' completed all six blocks with 
reliable tines were considered. A prolonged conputer failure during the 
evaluation period rendered nany of the evaluation group students' times 
unreliable and substantially reduced the available n. For the remaining 
75 evaluation qroup students, the nean course completion time was 136.13 
hours (21^.69 six-hour days), a reduction of 17. .iO hours (2.uu days) from 
the control group mean, an overall reduction of ,11.2 percent {F(1,1J9) = 
13.44-j, p = .001). 

A reasonable question is the extent to which block time reductions 
were attributable to increased homework. The percentage of lessons 
marked as homework actually declined from 8.57 percent for the control 
group to 4.00 percent for the evaluation group. Although at least part 
^ • 0^ this apparent reduction may have been due to changes in homework re- 

porting procedures, there is no evidence that the block tine reductions 
can be attributed to increased homework. Thus, the major factor con- 
tributing to the time reductions would appear to be increased student 
productivity during the normal snift. 

Implementation of Student Progress flanagement was found to have a 
Jiegative effect on first-attempt end-of-block test scores. As shown in 
Table 2, the evaluation group's scores tended to be two to four percent- 
age points lower than those of th> control group. F tests of the score 
differences oetween groups indicated that only the lilock 2 difference 
failed to at least approach significance at the .Jl level. 

It can be argued that, in a criterion-referenced testing environ- 
ment, a reduction in iioan block scores is not necessarily undesirable 
if tfie test failure rate is riot incrLMSed. The number of f -rst-attei.ipt 
block test failures in the evaluation and control groups were contrasted 
by cni-SMjarc tests. The percentage of failures, by block, in each 
group, the chi-square values, and the resultant significance levels are 
shown in Table 3. Although the percentage of failures was consistently 
higher under tlie evaluation condition, rhe differences were statistically 
significant (p < .Ob) in only two of the six blocks: blocks 1 and 3. 
It siiould be noted, however, that the total tine students spent in a 
block,- incliiding any time spent in remediation followinq a block test 
failure was, on tho average, less than the time to complete the block 
prior to iiiplenentdtion of Progress Management. 

^'.^.1.1.3 Student Attitude Oucstionnai re' Uata . Tiie same eight 
Student Attitude -f^uestionnai re .it^^ns used in formative evaluation were 
enployod to evaluate attitude- di ; rences boU/een the control and eval- 
uation (iroups. In chis case, data were available from both adminis- 
trations of thu questionnaire: following lilock 1 and at the end of the 
course, following lilock o. Item responses are arranged in a five point 
scale ranging iVoi.i "Strongly Jisagree" to "Stronnly Agree" and are 
scored such that thu most positive response is given a weight of five and 
the iiiost negative a v/eiqht of one. Tne items, median response fron 
both administrations, differences between the evaluation and control 
group riedians, the chi-square values (df = 4), and tho chi-Stiuare 
probability levels (if ;) < .10) are presented in Table 4. 

Tlio fir.t tiree itens [)ertain to students' (jorceptions of tlieir pace 
tlirou(,'h Lht.' first block (first Adi'nnist'-.iUc.n) or the course (second 
adi.iifnstrdtion). Tht; evaluation qnju'''s |)ati(.M'n of responses to the 
first iteii, weru si()nificantly iiior^ ri.'n.it.ive t i.in tiiose of die control 
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group at the end of Block 1, but the responses of the two groups were 
essentially equivalent at the end of the course. Whereas the median 
response of the control group declined over time, the evaluation group's 
median response increased as students gained experience with Progress 
Management. The evaluation group's responses to the remaining two items 
are all slightly more favorable than those of the control group but only 
one comparison was statistically significant. 

The next two items, pertaining to students' perceptions of the 
instructional methods and the career field, did not differentiate between 
groups. 

The final three items pertain to students' perceptions of tf."ir ' 
interactions with their instructors. In contrast to the formative 
evaluation, all three items demonstrated a slight positive shift, signi- 
ficant at the .Ub level in two of the six comparisons. 

2.4.1.1.4 Revisions to the Criterion Test . Bas 3d rn Phase I 
results, two items having very low error rates v^ere eliminated since 
they tested concepts covered by other items. Distractors were reworded 
for the remaining three items. The high error rate item, pertaining to 
the frequency with v;hich rate of progress data is provided to the 
student, was also eliminated since sti"ients quickly learn this infor- 
mation in the first few days of the course. Corresponding changes were 
made to the second form of the test and both forms were implemented with 
a more stringent passing criterion of 80 percent correct. 

2.4,1.2 Phase II Evaluation Procedures and Results . In the second 
Phase of summative evaluation, the effect of the Time Management Module 
per se was evaluated in the Inventory Management course. Two versions 
of the course were defined: an evaluation version, containing the Time 
Management Module as the student's first assignment after entering the 
learning center; and a control version in which students v;ere given a 
placebo handout explaining the operation of the SPMC. Student Progress 
Management and Individualized Instructional Assignment were in operation 
in both versions of the course. Course completion times had become some- 
what shorter and the Policy Function was raised to 95 percent. 

The evaluation and control versions of the course were each taught 
in two learning centers on each of two shifts. While random assignment 
of students to versions within learning centers woul-d have been a pre- 
ferable experimental design, it was thought that the continuing visible 
effects of the Course Completion Map would contaminate the control group. 
At the start of the f valuation period, all entering students were 
assigned to one of the two versions. It was intended that this pro- 
cedure would continue for at least 2 months. 

Intervening events, however, prevented completion of the full eval- 
uation plan. First, it was determined that instructors in one of the 
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evaluation group learning centers had misunderstood the procedures and 
had instructed students to skip the Time Management Module and continue 
with their normal course v/ork. These students were eliminated from the 
evaluation. To compensate for the reduced n_, the proportion of incoming 
students assigned to the evaluation group was raised to 65 percent. 

It was then learned that the AIS was to be Submitted to a second 
evaluation (Service Test) in which the second shift would be run with- 
out computer support. The Service Test Plan went into effect on the 
tv^entieth class day of the evaluatison period, preventing further data, 
collection for second shift students. Since very little data were 
available in the latter half of the course, data collection was continued 
one more v;eek for those first shift students who had entered the course 
prior to the Service Test. 

2.4.1.2.1 Module Performance Data . Reliable lesson times were 
available for only 63 of the students who completed the first block of 
instruction. For these students, the mean first-attempt lesson time was 
97.0 minutes (standard deviation of 31.9 minutes). 

Reliable, item-level criterion test data were available for 82 
evaluation group students: 44 on Form 1 of the test and 38 on Form 2. 
The Alpha reliability of Form 1 was .883 while Form 2 registered an 
Alpha of .903. This increased reliability was, however, obtained at the 
cost of increased difficulty. The mean percentages correct for the two 
forms were 74. b and o8.8, respectively. Further investigation revealed 
that these low means were largely attributable to the two performance 
items not being marked by the instructors for almost half of the 
students. If these missing scores had been present, the means for the 
two forms would have been 80.9 and 75.1 percent, respectively. Form 2 
of the test was slightly more difficult. Given the test criterion of 30 
percent correct, the first attempt failuve rate would have been quite 
high even if the missing perfonaance items had been marked. 

2.4.1.2.2 Block Conpletion Time and Score Data . A total of 79 
evaluation and 79 control group students began and completed the first 
block of the course during the evaluation period. The number completing 
each successive block declined, with only 16 evaluation and 28 control 
group students completing the full six blocks of the course. Students 
who completed the latter blocks tended, of course, to be faster work rs, 
regardless of treatment group. 

Students were found to be unevenly balanced across the evaluation 
and control groups with respect to predicted course completion times. 
While this suggested analysis of covariance, a question of interest 
concerned whether the presence of the Time Management Module had 
differential effects on the rates of students predicted to have fast 
course completion tines as compared with those predicted to complete the 
course more slowly. That is, was there an interaction between predicted 
course completion time and the presence/absence of the module? To test 



for such interactions, a Rair of linear models (Ward & Jennings, 1973) 
was defined for each block in which the criterion variable was block 
completion time (block elapsed time less absence time). The first pre- 
dictor variable was predicted course completion time, and the second 
predictor variable was a binary vector representing the presence or 
absence of the lime Management ttodule. For the "full" models, both 
the intercepts and slopes of the two regression lines (representing the 
evaluation and control groups) were allowed to vary between groups while, 
in the restricted models, only the intercepts were allowed to vary. Com- 
parisons of the error sums of squares of the two types of models, by 
meins of an F test, is a test for homogeneity of regression. That is, 
a' significant F would indicate that the slopes of the two regression 
lilies were not parallel. 

no significant (p < .05) interactions were found for any of the six 
blocks. In fact, five of the six F values obtained were less than 1.0. 
Since the assumption of homogeneity of regression was met, analysis ^f 
covariance was employed to evaluate the effect of the Time Management. 
Module on block completion times and scores. 

The means and standard deviations of the evaluation and control 
group block completion times (adjusted for differences in the covariable 
of predicted course completion time) and the raw and percentage differ- 
ences between groups are shown in Table 5. Since so few students com- 
pleted the full course during the evaluation period, cumulative times 
are also shown for students completing the third, fourth, fifth, and 
sixth blocks. 

As shown in Table 5, reliable B^ock 1 completion times were avail- 
able for 77 evaluation group students and 76 control group students. The 
adjusted mean block time for the evaluation group was 19.51 hours, not 
significantly different from the mean (19.54 hours) of the control group. 
This indicates that evaluation group students had regamed the time re- 
quired for the Time Management Module itself by the end of the first 
block. 

For Blocks 2 and 3, the mean block completion times of the eval- 
uation group were approximately nine percent shorter than those of the 
control group and both comparisons approached significance at the .05 
level: Block 2 (F(l,122) = 3.864, p = .052); Block 3 (F(l,98) = 3.483, 
p = .065). The fact that these relatively large differences failed to 
achieve significance was due primarily to the high variability of the 
block completion times. 

The Block 4 ana 5 comparisons indicate even larger time savinqs, 
almost 15 percent, attributable to the Tine Mananement Modulo, both of 
\/hich achieved significance: Block 4 (F(l,73) = 6.079, p = .016); Block 
5 (F(1.53) = 4.490, p = .039). 

The apparent time savings attirubtabie to the module in the sixth 
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TABLE 5 



CLOCK TIMES FOR PHASE II 
SUMMATIVE EVALUATION AND CONTROL GROUPS 



KtKLtNT 



WADTARI F 
VMKiMULL 


N 


EVALUATION 

Y 
A 




N 


CONTROL 
J 


SO 


EVALUATION 
- CONTROL 


AGE 
CHANGE 


DiocK 1 1 ime 


77 


1Q m 




76 


19 54 


6.29 


- 0.03 


- 0.15 


DiocK d 1 ime 




9"^ 77 


1 52 




26 10 


7.93 


- 2.33 


- 8.93 


Block 3 Time 


39 


32.11 


10.26 


62 


35.39 


11.01 


- 3.28 


- 9.27 


Block 4 Time 


28 


18.45 


5.76 


48 


21.70 


5.64 


- 3.25 


-14.98 


Block 5 Time 


20 


19.01 


5.91 


36 


22.36 


5.96 


- 3.35 


-14.98 


Block 6 Time 


16 


11.75 


2.63 


27 


15.12 


4.72 


- 3.37 


-22.29 


Total Time, 
Blocks 1-/. 


34 


74.11 


23.13 


53 


78.26 


21.77 


- 4.15 


- 5.30 


Blocks >4 


21 


86.10 


24.64 


38 


95.54 


25.23 


- 9.44 '■ 


- 9.88 


Blocks 1-5 


13 


95.19 


26.69 


25 


103.94 


25.52 


-13.75 


-12.62 


Blocks 1-6 


10 


101.22 


18.23 


19 


113.76 


23.64 


-17.54 


-14.77 


a. Tines, in hours, adjusted for differences in predicted 
tine. 


course completion 



block exceeded 20 percent but the number of students coiiipleting this 
block \ntli reliable times was so small as to make the comparison suspect. 
Despite the small n, the difference between the adjusted means was si(im- 
ficant (F = 0.674. df = 1/40. p = .014). 

Since the Phase I and II evaluation groups both received the sam'j 
treatment, one might expect the block times to be similar while, in fact, 
the Phase II evaluation .group times were consistently shorter. Two 
factors contributed to this difference: Course completion times tended 
to become shorter during the period between the two comparisons, and 
since the Phase II evaluation period was only half as long as the Phase 
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I period, data in the later blocks of the Phase II comparison tended to 
be from faster students. 

Cumulative times were obtained for those students who had reliable 
block completion times on the first through third, fourth, fi-th, and 
sixth blocks. These values, adjusted for differences in the covariable, 
are shown in the lower half of Table 5. Thirty-four evaluation and 53 
control group students completed the first three blocks with reliable 
times on all three blocks. The mean of the evaluation group was 4.15 
hours less than that of the control group, a difference Which was not 
statistically significant. By the end of the fifth block, the number 
of students having become quite small, the 13.75 hour savings did at 
least approach significance (F(l,35) = 3.954, p = .055). Finally, for 
those students with reliable times on all six blocks, the apparent 
advantage for the evaluation group was 17.54 hours (F(l,26) = 5.431, 
p = .023). Despite the several comparisons which failed to achieve 
statistical significance, trie consistency v^ith which the control group 
means exceeded those of the evaluation group strongly suggests that the 
Time Management Module did indeed have the effect of decreasing block 
completion times. 

The means and standard 'deviations of the six block test scores, 
adjusted for differences in the covariable, are presented in Table o. 
As is shown by the right-most column, there was no consistent pattern of 
differences between the two groups. Further, none of the differences 
were statistically significant as evaluated by analysis of covariance. 
The number of first-attempt evaluation and control group block test 
failures were also compared by means of Chi Square tests. None of the 
six comparisons approached statistical significance. It can be con- 
cluded that the time savings attributable to the Time Management Module 
were not achieved at the cost of increased block test failures. 

A comparison of the score means shown in Table 6 with those obtained 
during the Phase I evaluation suggests that the negative effects observed 
when Student Progress Management was first implemented were only tem- 
porary. The across-block, unweighted mean score of the Phase I control 
group was 81.92 while that of the Phase I evaluation group was 79.12. 
The unweighted mean of the Phase II evaluation group, 81.95, returned 
to the level observed prior to implementation of Progress Management. 

2.4.1.2.3 Student Attitude Questionnaire Data . The same eight 
Student Student Attitude Questionnaire items (see Table 4) wore again 
employed to evaluate differences in attitudes attributable to the Time 
Management Module, hor the first three items, pertaining to students 
perceptions of their pace through the course, only one of the six chi- 
square comparisons indicated a significant difference (p = .037) between 
the responses (on a 5-point, bipolar scale) of the evaluation and control 
group. Evaluation, group students more strongly disagreed witfi the state- 
ment "since Denver was such a nice area, I was not in any hurry to finish 
the c^ourse," than did control group students but only on the first 
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TABLE 6 

^ND-OF-BLOCK TEST SCORES FOR PHASE II 
SUMMATIVE EVALUATION AND CONTROL GROUPS 







EVALUATION 






CONTROL 




EVALUATION 


VARIABLE 


N 


. X 


SD 


N 


X 


SD 


- CONTROL 


Block 1 Score^ 


■79 


82.7 


9.72 ■ 


hi 


81.3 


10.58 


+ l;4 


Block 2 Score 


63 


81.3 


13.29 


79 


81.3 


11.47 


0.0 


Block 3 Scr-^e 


43 


80.0 


9.83 


64 


76.9 


10.70 


* 

+ 3.1 


Block 4 Score 


31 


82.5 


9.12 


51 


82.6 


10.41 


- 0.1 


Block 5 Score ' 


22 


81.1 


10.85 


39 


81.7 


10.64 


- 0.6 


Block 6 Score 


16 


84.1 


8.46 


28 


85.4 


8.20 


- 1.3 



a. Scores, shown as percentages, adjusted for differences in predicted 
course completion times. 



(Block 1) administration of the scale. The effects of the Time Manage- 
ment ftodule in this area were apparently only slight and diminished over 
time. 

The onl} . er^tem which approacntj significance (p = .060) was for 
the end-of-cou . administration of the statement "the instructors helped 
me and encouraged me to do well," on which the responses oV the eval- 
uation group were more positive than those of the control. 

Soon after the Progress Management Component had been implemented, 
20 items were added to the Attitude Quest ionnci ire dealing specifically 
with Student Progress Management. A comparison of the evaluation and 
control group responses to these items indicated that of the 40 r.hi- 
square comparisons only two (the number to be expec*'"'^ by chance) 
registered significant differences at the .05 level, uvaluatlon group 
students more strongly agreed with the statement "it was difficult con- 
centrating when I knew I was behind (target)," than did control group 
students on both thp Block 1 and end-of-course administrations of the 
questionnaire. This result could be Interpreted as either (a) the Timr 
Management Module Course Completion Map ma) have ncreased litudents' 
anxiety when they were behind target or (b) the control group students 
attached little importance to the target rate. 
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3.0 ST UDY miLS TRAINING PACKAGE 

3.1 Problem Definition 

Following the development and implementation of the Orientation to 
OMI and Time Management Modules, It became apparent from Instructor 
connents and observations that another area of student problems In the 
AIS environment was the lack of appropriate study skills. To further 
define specific skills deficiencies, a second series of interviews were 
conducted with personnel from the IM and MF courses. These Interviens 
were conducted during the formative evaluation of the Orientation 
Module, and four categories of personnel were interviewed: current 
students, students eliminated from their course, course Instructors, and 
course supervisor*. 

. Current stuojnts interviewed were members of one of four subgroups: 
(a) Experienced-^ood, (b) Experienced-Poor, (c) Na,1ve-Good, and (d) 
Naive-Poor. Experienced students were those who had finished all but two 
blocks of their course, with good students being differentiated f.'om poor 
on the basis of whether they were at least 2 days ahead of their Target 
Completion Date and had at 'least an 80% grade point'average versus being 
2 days or more behind their Target Completion Date and having a grade 
point average of less than 80%. Naive students were those. who were 
still working in the Mrst block of their course and were predicted to 
take less (good group) versus more (poor group) than the mean completion 
time to finish the course. A total of seven students were interviewed 
in, each of these four. groups, as well as eight Instructors, four super- 
visors, and two eliminees. 

Each interview lasted approximately 60 minutes, with about one- 
fourth of this time being devoted to the interviewee reading the 
Orientation Module. In the "Pre^Module" section of the interview, nine 
questions concerning students' feelings about being back in school, 
^tb&if^ study habits and their experience with individualized instruction 
were used. The results of the student responses to these questions are 
summarized in Table 7 ano the results of the Instructor and supervisor 
responses are summarized in Table 8. 

The following conclusions were drawn from these interviews. 

1. Although most students like the idea of taking responsibility 
for their own learning, most are generally not prepared to handle the 
responsibilities of individualized instruction, according to their in- 
structors; hence, there is a need for some type of remedial training in 
study skills and self-discipline. 

2. Since staff personnel indicated that they spend a major portion 
of their time with students who are having difficulty with their courses, 
these personnel would be likely to benefit from some training in study 
skills remediation approcches and general diagnostic and counseling skills. 
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TABLE 7 

Student Opinions of Their Study Skills 



QUESTIONS 



How do you feel about being back 
in school/about school in 
general? 



STUDENT RESPONSES 



I 



1. 



2. How would you rate your study 
skills? 



3. Have you ever had an individualized 
course before this one, and if so, 
what was your opinion of this 
experience? 

4. How do you feel about taking 
. responsibility for your own 

learning? 

^5. How would you feel if grades were 
eliminated and the course were 
merely pass/fail? 

*6. Do you plan your study time? 

*7. Do you use the objectives and 
the embedded questions? 

Do you use them the way the 
text suggests? 

How do you feel about the method 
the text suggests? 



Naive-Good students had th "owest. opinion of school; Naive-Poor 
students had the highest opinion. Good students had a lower 
opinion of school than Poor students and Naive students had a slightly 
better opinion of school than Experienced students. 

A majority of both Good and Poor students rated their study skills as 
average. About one- fifth of both the Good students and the Poor 
students rated their study skills as good. Several of the Poor 
students, but none of the Good students, rated their study skills as 
poor. 

Four of the 14 Good students had previous experience with individ- 
ualization, and one of the 14 Poor students had this type of 
experience. All of the students who reported prior individualiza- 
tion experience rated it positively. 

A majority of all students liked this feature, less than one fifth 
were ambivalent and none reported a dislike of this individualization 
characteristic. 

A majority of the Experienced-Poor students and all of the Experienced- 
Good students from the IM school indicated that they would dislike such 
a change, whereas both of the Experienced-Good students from the MF 
school thought they would like a pass/fail system." 

<\ majority of the Experienced-Good students reportedly planned their 
study time, whereas less the.i half of the Experienced-Poor did this. 

All of the Experienced students indicated that they used the 
objectives and the embedded questions. 

Less than half of the Experienced-Good students and less than one- 
third of the Experienced- Poor students reportedly used the, objectives 
and eiibedded questions in the manner sug.gested by the text. 

A majority of all Experienced students liked the method the text 
suggested. 



* Naive students were lot asked questions 5 through 9. 
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TABLE 8 

Staff Opinions of Student Study Skills 



STAFF (INSTRUCTORS & SUPERVISORS) RESPONSES 



QUESTIONS 



1. How do students feel about being 
back in school? 

2. What kind of study habits do 
students have? 

3. About how many students have had 
experience with individualized 
instruction before coming to this 
course? 

4. How do students feel about taking 
responsibility for their own 
learning? 

5. How would you feel if grades were 
eliminated and the course were 
merely pass/fail? 



6. Do students plan their study time? 



7. Do students use the objectives and 
embedded questions? 

b. Do they use them the way the text 
suggests? 

9. How do you feel about the method 
the text suggests'' 



A n^ajority of the staff believed that students either liked being 
back in school or were ambivalent about this situation; 

A majority of the staff indicated that most students have poor 
study habits. 

A majority of the stiff ndicated that, for most students, this CMI 
course is their first experience with individualized instruction. 



A majority of the staff stated that most students either do not know 
how or do not want t-^ take responsibility for their own learning. 

Slightly less than half the instructors felt that changing the course 
to a pass/fail system would be desirable. None of the supervisors 
agreed with this opinion although a majority of them were ambivalpnt 
on this subject. Slightly less than half the instructors felt t.iat 
such a change would be undesirable. 

Two-thirds of the instructors believed that students do not plan their 
study time, whereas a majority of the supervisors believed that 
students "sometimes" plan their study time. 

A majority of the instructors thought that students "sometimes" use 
the objectives and the embedded questions. Supervisors were evenly 
divided between "usually" and "sometimes." 

A majority of the instructors thought students "sometimes" used the 
objectives and embedded questions the way the text suggested, but 
supervisors were ever, divided between "sometimes" and "rarely." 

A majority of the staff liked the method suggested by the text for 
using objectives and embedded questions. 

w 
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3, Since most students indicated that they did not dislike school, 
it is likely that providing them with specific study skills training 
would improve their performance in the course. Thus, the interviews in- 
dicated the need for both student study skills training and instructor 
training in the diagnosis and remediation of student study skills pro- 
b1 ems . 

3.2 Module Design and Development 

The detailed rationale and procedures used in the design and 
development of the Study Skills Package are presented in AFHRL-TR-79-43 
That report also contains copies of the instructional materials and all 
evaluation instruments. Consequently, this section will provide only 
brief descriptions of the materials in the Study Skills Package (i.e., 
the four Study Skills Modules, the Study Skills Questionnaire, and the 
Instructor Orientation and Training Workshops). 

3.2.1 Study Skills Modules . The student and "'nstructor interview 
results indicated that many military trainees do not possess the study 
skills which are vitul to success in a CMI environment. Furthermore, 
those trainees who do possess such skills seem to have difficulty trans- 
ferring these capabilities from their prior lock-step experience to 
their present CMI experience. Hence, there appears to be a definite 
requirement for teaching students how to transfer appropriate learning 
strategies and skills from their past experiences to a new CMI environ- 
ment and how to improve upon or add to their study skills repertoire. 

It was hypothesized that a set of consumable modules (modules which 
students can keep) dealing with various study skills would be an appro- 
priate solution to the foregoing problem. The decision to make tne 
modules consumable was based on the idea that students often need to 
return to new material to freshen their memories and clarify confusing 
or difficult areas. In addition, since the use of good study skills is 
an on-going requirement in the course, it was judged desirable to give 
students a set of materials to which they could refer whenever it was 
necessary. 

The major design goal of the Study Skills Modules was to provide 
poorly skilled students with appropriate skills for improving their 
performance in the course and reduce the amount of time required to 
complete. The four study skills areas which were isolated as being the 
most troublesome were (a) Reading Comprehension, (b) Memorization, (c) 
Test Taking, and (d) Concentration Management. A set of materials was 
written for each of these four areas and packaged individually, so ti- 
a student could receive any or all of the packages as deemed necesser 
an instructor. 

3.2.1.1 Module Objective s. The objectives for the Reading Com- 
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prehension module were as follows: (a) You (the student) will be able 
to use the Questionning method of studying; (b) You will be able to use 
the Network method of studying; (c) You will be able to use the Problem 
Solving method of studying; and (d) You will be able to determine which 
of these three reading methods is best for you and to use this method 
when you are studying. . 

The Memorization module had one objective: You (the stuaenn} will 
be able to increase your skills in memorizing Information by using 
mnemonics such as elaboration, mental pictures and qrouping. 

The six objectives of the Test Taking module, entitled Test- 
Wiseness, were as follows: (a) You (the student) will be able to use 
methods for completing tests on time without giving up any accuracy; (b) 
You will be able to carefully read and understand test directions and 
test questions; (c) You will be able to use good methods for figuring 
out how and when to guess on test questions; (d) You will be able to use 
logical reasoning to answer test questions when you are not sure of the 
correct answer; (e) You w'll be able to understand that test anxiety is 
a habit, a hard-to-control feeling of ft .r of tests and thinking about 
tests; and (f) You will be able to use specific methods for controlling 
most of the test anxiety which you might have when you sit down to take 
a test. 

The two objectives of the Concentration Management module were: (a) 
You (the student) will be able to identify the difference between good 
study moods and bad study moods, and (b) You will be able to use proper 
methods for changing your bad study moods into good study moods. 

3.2.1.2 Instructional Strategies and Procedures . The primary in- 
structional strategies incorporated into the four Study Skills Modules 
were active information processing techniques and positive self-talk 
behavioral control techniques. In the Reading Comprehension Module, the 
three recommended reading methods required students to not only become 
actively involved in the material, but to also make changes to that 
material so as to personalize it and integrate the information with 
existing cognitive structures. SimiUrly, the Memorization Module intro- 
duced the concept of mnemonics, or memory aids, as methods for personal- 
izing information and making it more meaningful. Both the Test Taking 
Module and Concentration Management Module utilized the concept of 
appropriate self- talk as a method for reducing test anxiety, combatting 
boredom or anxiety when trying to study, and maintaining a good study 
mood. 



3.2.1.3 Module Format and Evaluation Instruments . The four Study 
SkiKs Modules were written In the same narrative style as the previous 
two modules; however, the use of cartoons and diagrams was significantly 
reduced due to the le» gth of each module. Each module was bound separ- 
ately as a consumable 3 1/2 by 11 inch illustrated text. 
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The evaluation instruments for the Reading Comprehension Module in- 
cluded worksheets and practice exercises for the three reading methods 
introduced. The Memorization Module used examples and practice exercises 
for each of the three types of mnemonic techniques (elaboration, mental 
pictures, grouping). A 20-item self test, followed by a test key and 
descriptions of how students could have used the test-taking skills pre- 
sented in the Test Taking Module to figure out the answer to the self- 
test questions if they had been unsure, was the evaluation instrument 
used in the Test Taking Module. Finally, the Concentration Management 
Module included several worksheets for students to use in recording what 
made them lose their concentration, how they felt about this loss of 
concentration, what they said to themselves, whether or not it was pro- 
ductive, and what they could have said or done that would have been more 
productive. In addition, both the Test Taking Module and the Concen- 
tration Management Module included a Slow Deep Breathing exercisp tn 
assist students in obtaining a "relaxed yet alert" sta'^e of mint (oiies 
of all these materials can be found in AFHRL-TR-79-43. 

3.2.2 Study Skills Questionnaire . In order to assist instructors 
in identifying students in need of study skills remediation, a Study 
Skills Questionnaire was written to assess each student's opinion of his 
or her study skills in the four selected areas (Reading Comprehension, 
Memorization, Test Taking, Concentration Management). All students were 
to be given the Questionnaire after completing the first block and before 
beginning the last block of the course. The results of the first admin- 
istration were intended to provide a preintervention measure and diag- 
nostic information for instructors as to the type of study skills 
remediation that would be most appropriate for a particular student. The 
results of the s.econd administration were intended to provide a post- 
intervention measure which, when combined with information on student 
performance in the course, could be used to evaluate the effectiveness 
of the Study Skills Modules. 

3.2.2.1 Questionnaire Format . The Study Skills Questionnaire 
originally consisted of a total of 50 items which were divided among 
four subscales. There were 15 items in the Reading Comprehension sub- 
scale, 12 items each in the Memorization and Concentration Management 
subscales, and 11 items in the Test Taking subscale. Students were 
asked to respond to each item by selecting one of three to five options 
which either asked them to rate their skills compared with other students 
or to select one of several study techniques which most matched their 
normal study methods. A copy of the original Questionnaire can be 
found in AFHRL-TR-79- 

3.2.2.2 Questionnaire Scoring Procedu res. For both administrations 
of the Study Skills Questionnaire, students in the four AIS courses 

were asked to record their responses on an AIS Generalized Answer Sheet. 
Five variables were defined tj the AIS data base for the scoring and 
recording of Questionnaire data: (a) a total Study Skills Questionnaire 
score, (b) a Reading Comprenension score, (c) a Memorization score, (d) 
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a Test Taking score, and (e) a Concentration Management score. The 
corresponding variables were defined as SSQUES, READCM, MEMORY, TSTTAK, 
and CONMGT to the AIS data base. All answer sheets were processed 
through AIS management terminals. 

3.2.3 Instructor Orientation and Training . The major goal of the 
instructor package was to provide instructors with appropriate tools to 
help them become efficient diagnosticians, tutors, and/or remediators 
of student study skills deficiencies, thereby enabling them to help 
students become efficient learners. The Study Skills Modules were used 
in the Instructor Orientation and Training Workshops as a primary focus 
of the CMI student-instructor interaction. In addition, a purpose of 
this package was to provide instructors with specific tools which they 
cojld use to define their counselor role in a CMI environment. 

3.2.3.1 Instructional Strategies and Procedures . Given that the 
Study Skills Package placed heavy emphasis on the instructor's role in 
(a) diagnosing student study skills problems via the Study Skills 
Questionnaire and (b) facilitating remediation of problem areas through 
assignment of the Study Skills Modules, volunteer inr,tructors were 
given intensive training in the skills requ»red in a 6-hour workshop. 
To elicit support for the program and obtain instructor volunteers for 
the workshop training, the investigator;, briefed instructors and super- 
visors in all four AIS courses on the purpose of the Study Skills 
Package and its various components (remediation modules, questionnaire, 
instructor train i^^^ji} . 

The basic approach in the workshop training was to provide in::truc- 
tors with an opportunity to analyze, dis'-.itss, and i^ractice the techniques 
presented in each of the f ur Study Skill' Modules. Jn addition, the 
workshops were derigned to train instructors in basic problei striving, 
diagnostic, and communication fkilU (e.g., listening, probing), to 
practice these skills in role plavinq '^xercises, -ind to provide follow- 
up sharing of successes and/or problems in impleriientinn their new 
diagnostic and tutorial skills, 

3.2.3.2 Trainirg Format and Ev aludtion ^ nstru l ent^.. The Instructor 
Workshops were divided into thrue Z hour sessions. The "first session 
was devoted to explaining, discussing and practicing the use of the study 
techniques described in thi four Study Skills l>k)du]es. The -/^cond 
session focused on problem solving skiUc, diagnostic stratecjie., a d 
remediation procedure; including some training ir; listening an J probing 
skills. These two session, were held on conser tive days, iid the thi d 
session was held approximately 1 week later, hri third ' -snion w s 
designed to give instructors a chance to diiicuss ary pi iems c lifti- 
culties they had experieiiced in using the nBtoriai id pr lurob in 
their classrooms and to exchange information, sol' .>iis sugqestio s, and 
interesting case histories, 

Following the seccnd s(ssicwi of th^ in.t*ur >» Work hop, it was the 



instructor's responsibility to decide which students should receive wi. ch 
of the Study Skills Modules. When an instructor identified a student who 
was behind schedule or who had failed a block test, the instructor was to 
determine if the student's problem was in the area of poor study skills 
and, if so, to select the appropriate type of study skills remediation. 
Although these decisions could have been determined statistically, by 
other personnel, or even by the students themselves, it was hypothesized 
that giving instructors this responsibility was vital to the development 
of appropriate instructor roles. An instructor critique form was 
designed to assess instructor reactions to and comments about the work- 
shops. This critique form contained 19 forced-choice ratings and one 
open-ended question. Copies of the materials used in the workshops and 
the critique form can be found in AFHRL-TR-79- 43. 

3.3 Formative Evaluation 

As with the Orientation to CMI and Time Management Modules, the 
materials developed as part of the Study Skills Package were evaluated 
in two phases during formative evaluation. The first phase was a small- 
grnup tryout d the second phase was a more extensive operational 
tryiut. The \ jsults of these evaluations are described separately for 
the Study ".kills Modules, the Study Skills Questionnaire, and the In- 
struc or Orientation and Training. Since extensive descriptions of the 
procedures and results of formative evaluation activities are described 
in AFMi -rR-/9- . only summaries of these orocedures and results will 
bo presented here. 

J.J.I Study Skil ls Modules . Small group tryouts of the four Study 
Skills Mo(J';les employed instructors from the four AIS courses, rather 
than students, for three main reasons: 

It wds hypothesized that if instructors were included in the 
fon..rt».. /o evaluation of these materials, the • would be more accepting of 

t.iem diriny suh' njuent evaluations. 

Z, Since ir nning in study skills te... iques was part of the 
hr.trurtor Ot ietitalion and Training, it was thought that instructor par- 
..icipaMon would greater if their conments and suggestions were given 
V(>1 idi ty and valuo. 

J Mjch of tiM. Kiat.enal vvhich w^is used in these modules had already 
bfcn suhj.'tteo tu stmJeiit ^^valu.n ion by other researchers (e.g., 
iMr.'.ereau, :)ote 12i Wpinvf^m, 'Aotn I'j' Woodley, Note 16). 

Following investigator oric- inys ot -ach of the AIS courses, 
sLtudn, volunteers wr--r as. i d to rnad and complete a 12-item que: .on- 
naire on any one of l,r Fnuv s-udy snils Kodules. A list was al l 
circulated F r interest* i rtru tors to siqn up for participation in 
the subsequent Instructor ^^ st,. s One week later, the questionnaires 
and instructor comnents were dr. , vl'h wi r.h those instructors who had 
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yolunteared to participate in the evaluation, and requests for additional 
instructor volunteers for the workshops were circulated. 

Major findings from this small-group tryout were (a) instructors 
generally liked the modules and believed that they would be useful to at 
least some of their students, (b) instructors generally believed that 
students could take any of the modules by themselves, although they also 
te It instructors could give students tutorial assistance in the study 
Skills techniques if necessary, and (c) a majority of the instructors 
reit that the modules were written at too difficult a level for most 
students and, therefore, needed to be revised to reduce the use of 
technical terms and to clarify the procedures described for the various 
study skills techniques. Based on these results, all modules werp 
revised and generally simplified, with the most extensive changes being 
made to the Memorization Module. 

Th3 revised Siudy Skills Modules were subjected to an operational 
tryout in the IM, MF, and WM courses, in that instructor volunteers for 
workshop training were obtained from operational AIS blocks in these 
courses. (The instructor volunteers from the PME course were from non- 
AIS blocks). The operational evaluation time period was approximately 
10 weeks, during which time the instructors identified students in need 
of one or more of the Study Skills Modules and assigned them to these 
modules either during the regular shift or as homework. The four Study 
Skills Modules were identified as lessons in the overall course 
hierarchy, a procedure which provided for the identification of which 
students per course were assigned which Study Skills Modules, without 
having the time spent on the module(s) count against their course com- 
pletion time. This procedure was felt to be necessary to avoid the per- 
ception by students of any negative consequences being associated with 
study skills remediation. 

The questions of primary interest during this operational tryout 
were whether those students who received one or more of the Study Skills 
Modules would (a) increase their scores on the Study Skills Questionnaire 
from the initial to end-of-course measurement periods, and (b) would 
improve their course performance (either times-to-complete or scores) 
following remedial study skills training. Thus, the evaluation employed 
a within-student design in which preliminary data on the effectiveness of 
the modules for improving performance were collected in the three AIS 
courses. A question of tangential interest was the number of students 
per course the instructor workshop ['articipants would identify for 
study skills remediation. 

The number of IM, MF, and WM students who -ere identified for and 
received one or more of the modules during the evaluation period are 
shown in Table 9. These data do not, however, reflect a completely 
accurate picture of the extent to which the instructors who participated 
in the workshops assigned these study skills materials. It was learned 
from the instructors and data clerks in the three courses that a number 
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TABLE 9 



Number of Students Receiving Study 
Skills Materials During Operational Tryouts 



COURSE 


TOTAL STUDENTS 
WHO rOMPL ETFD 
THE COURSE 


STUDENTS 
WITH COMPLETE 
PERFORMANCE 
DATA 


TOTAL WM 
STUDENTS STILL 

^ 1 Viyiall 1 «^ 1 Ababa 

IN COURSE 


WM STUDENTS 
WITH COMPLETE 
PERFORMANCE 
DATA 


IM 


7 


5 






MF 


2 


0 






MM 


2 


1 


10 


5 



of instructors had picked up materials for assignment to students, but 
had not identified the students to the AIS data base. A conservative 
estimate of the additional students is 10 in the IM course, 5 in the MF 
course, and 25 in the WM course. 

Given the small number of students per course with sufficient per- 
formance data for analysis, individual students' pre/post block time and 
score data are reported in Table 10. These data are shown as standard- 
ized scores to allc comparisons of performance changes across courses 
with variable block lenqths and block test difficulty levels. For each 
block, score and time means and standard deviations were computed for 
all students in each course during the evaluation period. The appropri- 
ate mean was subtracted from each student's (time or score) performance 
value for each block. The resultant value was then divided by the 
appropriate standard deviation. This procedure yielded an index for 
each student for each blotk which represented his or her performance 
relative to all other students. Finally, for each student, an average 
standard score was computed for both the student's pretreatment period 
and for the po<.t- treatment period. It is these pre- and post-treatment 
scores which are reported in Table 10. Table 10 also indicates, by 
student, the particular Study Skills Module(s) which are assigned and 
reports the uve»all average pre/post standard scores per course. 

As can be seen in Table 10, all five IM students improved their 
block times (a reduction in the standard score values) from pre to post 
study skills remediation, from an average of 2.86 standard deviations 
above the mean for pre- treatment, to an average of 1.30 standard 
deviations above the mean for post-treatment. Similarly, four of the 
six WM students improved their block times from pre to post study skills 
remediation, while two of the six students demonstrated some increase 
in block times. The average pre/post block time scores were 2.21 and 
1.24, respectively, indicating an overall improvement in WM student 
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TABLMO 



Standardized Pre/Post Study Skills Remediation 
Block Times and Scores for IM and WM Students 



Course/ 
Student 



Pre-Treatment 
Score 



Post-Treatment Modules 
Score Taken^ 



Block Time 

IM-1 
IM-2 
IM-3 
IM-4 
IM-5 
IM Average 

WM-1 
WM-2 
WM-3 
WM-4 
WM-b 
WM-6 
WM Average 

Overall Average 

Block Score 

IM-1 
IM-2 
IM-3 
IM-4 
IM-5 
IM Average 

^ WM-1 
WM-2 
WM-3 
WM-^ 
WM-5 
WM-6 
WM Average 

Overall Average 



1.77 
1.94 
2.75 
5.45 
2.40 
2.86 

2.48 
2.34 
1.89 
4.51 
.71 
1.30 
2.21 

2.54 



■1.12 
■1.14 
■1.88 
■2.24 
• .89 
•1.45 



-1.82 

- .11 
-2.43 

- .20 

- .10 

- .93 

-1.19 



1.57 
.09 
.89 
2.00 
1.96 
1.30 

-.84 
1.40 
1.22 
2.60 
1.42 
1.63 
1.24 

1.27 



.41 
.03 



RC, M, CM 
RC. M, CM 

RC 

RC. M, n, CM 
TT 



RC. M, TT. CM 

RC. M. TT. CM 

RC. M. TT. CM 

RC. M. n. CM 

RCj 
RC 



-.13 


RC. 


M. 


CM 




-.59 


RC. 


M, 


CM 




-.93 


RC 








-.29 


RC. 


M. 


TT. 


CM 


.25 


TT 








-.34- 










.95 


RC. 


M. 


TT. 


CM 


.30 


RC. 


M. 


TT. 


CM 


-.32 


RC. 


M. 


TT. 


CM 


.51 


RC 






.61 


RC 









RC = Reading Comprehension; M = Memorization; TT 
CM = Concentration Management 

No Pre/Post Score Data Avail ahle 



= Test Taking; 



ERIC 



-64- 



Hi 



block times of one standard deviation. It is also of interest that the 
WM students who were assigned all four Study Skills Modules all improved 
their block times, while the WM students who were assigned only the Read- 
ing Comprehension Module showed some increase in block times* It may be 
that increasing students' reading comprehension skills through the use 
of active information processing strategies increases their productive 
on-task time for WM course tasks, a possibility supported by the block 
score results shown on the bottom naif of Table 10. The overall improve- 
ment in student block times, however, across the IM and WM courses in- 
dicates that study skills remediation led to an improvement; of 2.54 to 
1.27. 

Pre/post block score changes for IM course students indicate tnat 
all five students, improved (an increase in standard scores) their block 
scores following study skills remediation. The average improvement was 
from -1.45 to -.34. All five WM students with pre/ post block scores also 
improved their performance following study skills remed ition, with the 
average chantje being from -.93 to .41. Thus, an improvement in block 
scores was found, even for those WM students who were assigned only the 
Reading Comprehension Module. It is interesting to no< . that even 
though two of the IM students were assigned only one Study Skills Module, 
they made both time and score improvements following remediation. This 
may have been due to an appropriate instructor match of materials to 
student needs. The overall improvement in student block scores across 
the IM and WM courses was from -1.19 to .03— again in excess of one 
standard deviation. 

Of those ' Ludent samples enumerated in Table 9, only two IM students 
had data on both the initial and end-of-course administrations of the 
Study Skills f]uestionnaire. The data for these students on the total 
scale (SSQUES. and four subscales (READCM, MEMORY, TSl.AK, CONMGT) are 
shown in Table 11, along with a listing of the particular Study Skills 
Modules these students were assigned during the course. It is of 
interest to note that the student who receiyed all four modules Remon- 
strated the greatest improvement in his assessirent of his study skills, 
whereas there was little change in the ratings of the student who 
received only the Reading Comprehension Module. 

Finally, it was of interest to examine the time spent on each of 
the Study Skills Modules. Given that, of the 11 students in the IM and 
WM courses with data available for analyses, only three IM and one WM 
student took their module(s) during the t«^aining shift, it is only 
possible tp suggest the time spent on the modules. Of the three IM 
students, times were available on the Reading Comprehension Module only, 
and these times-to-complete were 81, 109, and 86 minutes— a mean of 92 
minutes. The WM student took 275 minutes to complete the four Study 
Skills Modul'is. Assuming a liberal 35 J minutes for students to complete 
all four Study Skills Modules and a time savings of approximately one 
standard deviation for those students receiving '^tudy skills remediation, 
in the IM course (where the standard deviation ii. course completion time 



TABLE 11 



Changes in Student Scores on 
Initial and End-of^-Course Study Skills 
Questionnaire Administrations * 



COURSE/ 




INITIAI 


til J 


STUDENT 


SCALE 


sroRF 




IM-1 


SSQUES 




70 




READCM 


19 


21 




MEMORY 


18 


17 




' TSTTAK 


12 


15 




CONMGT 


20 


17 


IM-2 


SSQUES 


" 1 


68 




^PEADCM 


24 ' 


21 




MEMORY 


10 ' 


20 




TSTTAK 


. 02 


16 




CONMGT 


J6 


11 



MODULE(S)' 
■'KEN 



Re-iding 

Cor; 'prehension 



All 4 



1.^ 



is approximately 1600 minutes), a savings of approximately ''SO minutes 
out of a average completion time of approximately 7100 minutes (17.6 
)ercent/ would be realized for some percentage of the students. 
Simi 'arly..in the WM course (where the standard deviation in course com- 
tJletion time is approximately 2300 minutes), a savings of approximately 
19bU minutes out of an average completion time of approximately 11,700 
minutes (lb. 7 percent) would be realized for some percentage of the 
students. These results sug^st that the time required for study skills 
remediation compares favorably .wi th the resulting savings in completion . ' 
time. • • 

^.3.2 Study Skills Questionnaire . Small group and operational try- 
outs of the Study Skills Questionnaire were conducted at the same time 
as the srljall -group tryouts of the Study Skills Modules. During this 
time (d period of approximately 3 weekr.), ^he questionnaire was first 
revif.- .^d by instructors in the foui' A!S courses. The results of this 
review indicated that instructors were generally satisfied with the con- 
tent and format of the questionnaire. The questionnaire was then admin- 
istered to thoie students ^.n the IM, HF, WM, and PME courses who were 
beqinning the second block of their course or who were beginning the last 
course ck. 

The question of primary interest in the operational tryouts was 
wfi?'Jur the initial bO-item Study Skills Questionnaire demonstrated' 
satisfactory total sral^ and subscale reliability. Cronbach's alpha 
rt»l i.ibi 1 i ty coeffirient was calculated on each course's pre- and post- 
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questionnaire data via the AIS Test Item Evaluation (TIE) program. Un- 
fortunately, during this tryout period, student flow was low in all 
courses, and the sample sizes available fcr thp TIE analyses were 
sufficient for suggestive results only (i.e., number of pre-scale samples 
ranged, across courses, from 7 to 16 and the number of post-scale samples 
ranged from 7 to 22). Given these small samples, a conservative 
approach was adopted to revisions— to drop only these items which con- 
sistently demonstrated low item remainder correlations with the total 
scale or appropriate subscales across, all four of the AIS courses. This 
procedure resulted in deleting a total of 20 items from the question- 
naire. Eight items were retained in the Reading Comprehension subscale, 
seven items in the Memorization subscale, six items in the Test Taking 
subscale, and nine items in tne Concentration Management subscale (refer 
to AFHRL-TR-79-43 for a copy of the revised 30-item Study Skills 
Questionnaire). 

3.3.3 Instructor Orientation and Training . Four instructors from 
the MF course and four instructors from the WM course partic.pated in 
the small group tryouts of the Instructor Orientation and Training Work- 
shops. The critique results indicated that the instructors were 
generally satisfied with the composition and format of the workshops. 
Those activities which were most popular were the explanations of the 
study skills techniques presented in the four Study Skills Modules, and 
the diagnostic question handouts and explanations. Other activities 
rated highly were the exercise in listing internal and external clues to 
student study skill problems, the examples of when communication problems 
occur, the passage on Attending Behavior, and the discussion of re- 
flection and probing skills. 

Based on these findings, the workshop procedures and content were 
modified slightly, such that some examples were rjade more relevant and 
participative discussion was emphasizec' Additionally, it was decided 
to retain the Slow Deep Breathing exercise, even though several of the 
instructors had indicated feeling somewhat embarrassed about doing this 
e.-ercise in a group setting. It was deemed particularly important for 
instruccors to understand how th s technique worked so that they would 
be in a position to recommend it to s .id its experiencing test anxiety 
or having difficulty becoming relaxed white studying. 

The operational tryouts of th? Instructor Orientation and Training 
Workshops were conducted 2 weeks after the small group tryouts, and be- 
fore the operational tryouts of the Study Skills Modules and Question- 
iidire. Forty-seven instructors initially volunteered to participate in 
the workshops and four sections were created, with 11 instructors assign- 
ed to each section. Uue to summer leave and various job responsibilities, 
however, only half of these individuals actually participated: eight 
instructors each from the PME and IM courses and seven instructors from 
tho Wfl course. 

The results froin the instructor critique form indicated that in- 



-67- 

ERIC 



structors generally liked the workshop sessions and believed that they 
were useful to them as instructors. It also appears that the revisions 
made following the small group tryouts were appropriate since all of 
the workshop activities were rated as "particularly liked" by a majority 
of the instructors. A description of the final content and format of 
the workshops can be found in AFHRL-TR-79- 43. 

3.4 Summative Evaluation 

A second large scale tryout of components within the Study Skills 
Package wis restricted to the Study Skills Questionnaire due to time 
and resource cor.:.traints. 

3.4.1 Study Skills Questionnaire . The two questions of primary 
interest in the summative evaluation of the Study Skills Questionnaire 
were (a) whether the questionnaire demonstrated satisfactory reliability, 
as defined by an internal consistency measure and (b) whether it demon- 
strated acceptable validity in both a construct and predictive sense. To 
these ends, therefore, the revised 30-item questionnaire was implemented 
in all four AIS carrses for a period of approximately nine weeks. For 
the purposes of wlidating the questionnaire, data from the initial 
ad'»^ini strati on (I.e., end of first course block) are most relevant. 
Tins is because the questionnaire was intended at a diagnostic pre-course 
or pre-treatment self-appraisal of student study skills. Therefore, data 
on the questionnaire administered to AIS students at the end of their 
first course block were utilized to determine the rel. ability and 
validity of the Study Skills Questionnaire. 

3.4.1.1 Questionnaire R e liability Results . The means, standard 
deviations, and alpha reliability coefficients for the 30-item Study 
Skills Questionnaire (SSQUES), its 8-item Readin" Comprehension (READCM) 
subscale, 7- 'item Memorization (MEMORY) subscale, 6-ite.:i Test Taking 
(TSTTAK) subscale, and 9-item Concentration Management (CONMGT) subscale 
are reported in Table 12 for the Inventory Management (IM), Materiel 
Facilities (MF), Precision !^:asuring Equipment (PME), and Weapons 
Mechanic (WM) courses. The AIS Test Item Evaluation (TIE) program was 
used in the calculation of all reliability results. 

As Table 12 indicates, the alpha reliabilities of the SSQUES ranged 
from a low of .81 to a high of .95, indicating high internal consis- 
tencies for th<? total questionnaire acioss the four AIS courses. It 
should be noted that only in the WM course did the alphi* reliability 
of the total scale drop below .90. The reliability da .a reported in 
Table 12 also indicate that (a) the alpha reliability of the READCM 
subscale ranged from a low of .58 to e high of .87, (b) the MEMORY sub- 
scale reliability ranged from a low of .35 to a high of .75, (c) the 
alpha reliability of the TSTTAK subscale ranged from a low of .48 to a 
high of .84, and (d) the CONMGT subscale alpha reliability ranged from 
a low of .82 to a high of .88. Again, reliability coefficients from 
the MM course tended to be lower on all subscales than the other AIS 
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TABLE 12 

Means, Standard Deviations, and Alpha Reliabilltias 

of Study Skills Questionnaire Total Scale and 
Subicales Administered In Four AIS Courses 

Course ScaTi Score Range N Mian SD Alpha" 

IM SSQUES 30 - 120 313 81.0 17.5 .95 

READCM 8-32 313 22.9 5.1 .87 

MEMORY 7 - 23 313 1'7.3 4.1 .73 

TSTTAK 6-24 313 16.6 4.0 .82 

COUMGT 9 - 36 313 24.1 6.1 .88 

MF SSQUES 30 - 120 92 bl.7 16.1 .94 

READCM 8-32 92 23.2 4.5 .82 

MEMORY 7 - 28 92 16.7 4.0 .75 

TSTTAK 6-24 92 16.5 4.0 .84 

COi^MGT 9 - 36 92 25.3 5.8 .88 

PME SSQUES 30 - 120 79 82.6 13.7 M 

READCM 8-32 79 23.2 4.2 .80 

MEMORY 7-28 79 18.0 3.5 .bS 

TSTTAK 6-24 79 16.8 3.2 ./5 

CONMGT 9-36 79 24.6 5.1 .84 

WM SSQUES 30 - 120 297 85.1 9„3 .81 

READCM 8 - 32 2^:, ^4.1 2.3 .58 

MEMORY 7 - Zl> ^'97 lo.2 2.b .3f 

TSTTAK 6 - 297 18.0 Z.il .48 

CO.^MGT 9 " 3b 2^7 24. y 5.4 
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courses, although the CONMGT subscale demonstrated consistently high 
reliability across all four AIS courses. One possible explanation for 
the lower reliability coefficients in the WM course data may be the 
generally lower variability in these data. 

Item-remainder correlations of the individual SSQUES items, with 
both the total scale and the appropriate subscale, were determined 
separately for each AIS course. Tables 13 through 16 present the means, 
standard deviations and item- remainder correlations for the individual 
items for data from th2 IM, MF, PME, and WM courses, respectively. 

As indicated in Tables 13 through 20, five items deinonstrated con- 
sistently low item remainder correlations with their appropriate sub- 
scales across the four AIS courses. These items are: (a) Item 5 on 
the READCM scale, (b) Items 9, 12, and 14 on the MEMORY scale, and (c) 
Item 18 on the TSTTAK scale. These items are candidates for subsequent 
revision to increase the overall reliability of the Study Skills 
Questionnaire and of its subscales. 

3.4.1.2 Questionn a ire Validity Resul ts. The validity of the Study 
Skills Questionnaire was assessed in two ways. First, its construct 
validity was addressed by determining the extent to which the question- 
naire and its subscales demonstrated moderately high intercorrelations 
and consistent conceptual groupings across the four AIS courses. These 
construct validity questions were assessed by correlational and factor 
analyses, respectively. Second, its predictive validity was addressed 
by determining the extent to which the questionnaire and its subscales 
were predictive of student performance, in general, and of the perfor- 
mance of particular subgroups. These questions were assessed by dis- 
criminant analyses. Routines from the Statistical Package for the Social 
Sciences (SPSS; Nie, Hull, Jenkins, Steinbrenner, & Bent, 1975) were 
utilized iri the foregoing analyses. 

3.4.1.2.1 Construct Validity . As evidence of the construct 
validity of the SSQUES and its READCM, MEMORY, TSTTAK, and CONMGT suV 
S' ales, ritercorrelation matrices were calculated separately for student 
data on these measures from each of the four AIS courses. These matrices 
are shown in Tables 17 through 20 for the IM, MF, PME and WM courses, 
respectively. Tfie results generally indicate moderate to moderately 
high intercorrelations between the SSQUES and its subscales, suggesting 
that the four subscales of the SSQUES are within the same student 
characteristic variable domain. 

A further examination of the Questionnaire's construct validity was 
conducted via factor analyses of IM and WM course data. Only these two 
courses had sample sizes considered adequate for this statistical method- 
ology. Data from the SSQUES and four subscales were combined with 
student data on the set of affective and cognitive preassessment 
variables assessed ai the beginning o1 the courses. Descriptions of the 
variables include I in the IM troassessment battery are given in Table 21, 
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TABLE 13 



Means, Standard Deviations, and Item-Remainder 
Correlations for Study Skills Questionnaire 
Total Scale and Subscales In IN Course (N-313) 



Subscale/Item 


ncQii 


SC 


Item Remainder 
Total Subscale 


READCM 


1 




Q2 


.62 


.65 




2 




1 04 


.55 


.48 




0% 

3 


2.71 


.74 


77 


7n 




4 


2.69 


.80 


• 67 


• DO 




5 


2 91 


91 


CO 


cn 
• Do 




6 






• DC 


• DC) 




7 


9 77 

Cm i 0 


89 


71 
• /I 


• DU 




8 


9 77 


89 




• 66 


MEMORY 


9 


2.60 


1.03 


• 4i* 


• 03 




10 


2.41 


.78 


CO 


• Do 




11 




86 


70 


• DO 




12 




99 


• J<) 


9A 
• ^D 




13 




9B 


• 00 


An 

• HU 




14 


c • 


1 14 


oo 

• 07 


• 07 




15 


? 70 


75 


70 


Al 
• DO 


TSTTAK 


16 


9 R? 

CmOC 


78 


7*^ 


71 
• / 1 




17 




9B 


.61 


.51 




18 


2 59 


1.02 


.43 


.37 




19 


2.65 


.83 


.69 


.66 




20 


2.32 


.86 


.65 


.64 




21 


3.10 


1.05 


.69 


.65 


CONMGT 


22 


2.67 


.79 


.77 


.71 


23 


2.43 


.84 


.69 


.70 




24 


2.69 


.99 


.60 


.60 




25 


2.69 


.90 


.66 


.67 




26 


2.68 


.96 


.63 


.60 




27 


2.48 


.89 


.62 


.62 




28 


2.69 


.94 


.73 


.70 




29 


2.94 


1.03 


.59 


.57 




30 


2.86 


1.11 


.59 


.60 
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TABLE 14 



Means, Standard Deviations, and Item-Remainder 
Correlations for Study Skills Questionnaire 
Total Scale and Subscales In NF Course (N"92) 



Subscale 






SD 


Item Remainder 










Total 


Subscale 


READCM 


1 


2.72 


.86 


.60 


.57 




2 


3.14 


1.03 


.53 


.51 




3 


2.61 


.66 


.66 


.57 




4 


2 86 


79 


.58 


.55 




5 


2.83 


.82 


.52 


.51 




6 


3.30 


.92 


.60 


.55 




7 


2. BO 


.84 


.72 


.55 




8 


2.91 


.85 


.58 


.52 


MEMORY 


9 


2.53 


.93 


.54 


.24 




10 


2 34 




.62 


.55 




11 


2.86 


.76 


.74 


.65 




12 


1.93 


.97 


.34 


.33 




13 


1.97 


.88 


.47 


.60 




14 


2.46 


1.13 


.31 


.39 




15 


2.60 


.80 


.71 


.61 


TSTTAK 


16 


2.84 


.77 


.73 


.71 




17 


3.n 


.92 


.60 


.49 




18 


2.77 


.93 


.52 


.46 




19 


2.60 


.77 


.69 


.72 




20 


2.25 


.83 


.68 


.66 




21 


2.91 


1.08 


.65 


.70 


CONMGT 


22 


2.79 


.76 


.69 


.72 




23 


2.64 


.90 


.61 


.64 




24 


2.84 


.91 


.65 


.65 




25 


2.84 


.83 


.65 


.66 




26 


2.76 


.97 


.56 


.65 




27 


2.58 


.91 


.52 


.50 




28 


2.67 


.88 


.65 


.58 




29 


3.05 


.95 


.59 


.66 




30 


3.15 


.96 


.64 


.61 
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TABLE 15 



Means, Standard Deviations, and Item-Repalnd-r 
Correlations for Stucly Skills Questionnaire 
Total Scale and Subscales In PME Course (N-79) 



Subscale 



Item 



Mean 



SD 



Item Remainder 
Total Subscale 



READCM 



MEMORY 



TSTTAK 



C0NM6T 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

12 
13 
14 
15 

16 
17 
18 
19 
20 
21 

22 

23 
24 
25 
26 
27 
28 
29 
30 



2.34 
3.20 
2.70 
2.77 
2.65 
3.39 
2.73 
2.91 

2.34 
2.43 
2.86 
1.94 
2.68 
2.96 
2.81 

2.81 

3.28 

2.65 

2.82 

2.18 

3.10 

2.77 
2.58 
2.72 
2.70 
2.70 
2.47 
2.68 
3.05 
2.91 



.79 
.88 
.77 
.72 
.99 
.76 
.73 
.77 

1.04 
.80 
.76 

1.28 
.97 

1.01 
.68 

.62 
.88 
.82 
.66 
.78 
1.00 

.62 
.71 
.89 
.74 
.91 
.92 
.84 
.86 
1.12 



.51 

.58 

.65 

.60 

.40 

.67 

.60 

.63 

.48 
.49 
.58 
.20 
.28 
.28 
.62 

.61 

.59 

.28 

.58 

.40 

.61 

./7 

.63 

.44 

.60 

.61 

.56 

.71 

.63 

.48 



.53 

.51 

.57 

.50 

.38 

.65 

.49 

.57 

.35 
.51 
.46 

.06 
.03 
.24 
.57 

.56 
.48 
.25 
.56 
.50 
.67 

.71 

.59 

.33 

.61 

.55 

.57 

.64 

.60 

.52 
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TABLE 16 

Means, Standard Deviations, and Item-Remainder 
Correlations for Study Skills Questionnaire 
Total Scale and Subscales In WM Course (N-297) 



«;..K«r«i« T* Item Remainder 

^"^^g^^g Item K^an SD Total Subsr^ip 

«^ADCM 1 3.00 .69 .44 .46 

2 3.35 .73 .19 .23 

3 2.80 .56 .36 .29 

4 2.68 .67 .28 .22 

5 2.83 .70 .17 .15 
S 3.44 .69 .23 .26 
I 3.05 .70 .50 .29 
8 2.93 .74 .44 .39 

MEMORY 9 2.60 .88 .22 - ni 

10 2.49 .71 .38 :33 

]\ 3,05 .65 .62 .30 

12 2.21 .90 .14 .07 

13 2.12 .86 .04 .15 

14 2.81 .98 -.10 .05 

15 2.87 .52 .45 .37 



.27 



TSTTAK 16 3.04 .53 .30 

1^ 3.21 .74 .32 :i6 

18 2.77 .86 -.05 -.04 

19 2.90 .57 .53 .44 

20 2.67 .72 .47 .35 

21 3.38 .75 .55 .44 

C0NM6T 22 2.88 .58 .36 .18 

23 2.71 .77 .54 .62 

24 2.89 .78 .47 .53 

25 2.89 .77 .51 .60 

26 2.65 .90 .59 .62 

27 2.46 .91 .56 .57 

28 2.85 .83 .57 .56 

29 2.76 1.27 .42 .54 

30 2.80 1.32 .48 .60 




TABLE 17 



Intercorrelatlons of Initial Measure of Study Skills 
Questionnaire Variables in IM Course (N"267) 





READCM 


MEMORY 


TSTVAK 


CONMGT 


SSQUES 


READCM 


1.00 


.37 


.43 


.55 


.77 


MEMORY 




1.00 


.39 


.26 


.65 


TSTTAK 






1.00 


.41 


.72 


C0NM6T 








1.00 


.77 


SSQUES 










1.00 



All above correlations are significant at the p < .001 level. 



TABLE 18 

Intercorrelatlons of Initial Measure of Study 
Skills Questionnaire Variables in MF Cour&f (N-90) 





READCM 


MEMORY 


TSTTAK 


CONMGT 


SSQUES 


READCM 


1.00 


.40 


.4b 


.50 


.76 


MLMORY 




1.00 


.01 


.37 




TSTTAK 






1.00 


.41 


.77 


CONMGT 








1.00 


Jt 


ssqurs 










l.OU 



All above correlations are signifir. r.t at the p < .001 level. 
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TABLE 19 



Intercorrelatlons of Initial Measure of Study Skills 
Questionnaire Variables In PME Course (N-80) 





REAOCM 


MEMORY 


TSTTAK 


CONMGT 


SSQUES 


READCM 


1.00 


.47 


.38 


.67 


.84 


MEMORY 




1.00 


.23* 


.44 


.64 


TSTTAK 






1.00 


.46 


.64 


CONMGT 








1.00 ' 


.88 


SSQUES 










1.00 



* p < .05; all other correlations are significant at the p < .001 level. 



TABLE 20 

Intercorrelatlons of Initial Measure of Study Skills 
Questionnaire Variables In WM Course (N*271) 





READCM 


MEMORY 


TSTTAK 


CONMGT 


SSQUES 


KtADCM 


1.00 


.38 


.48 


.48 


.78 


MEMORY 




1.00 


.41 


.22* 


.62 


TSTTAK 






1.00 


.39 


.71 


CONMGT 








1.00 


.81 


SSQUES 










1.00 



* P < .01; all other correlations are significant at the p < .001 level. 




actor 



Test and Subscale Names 



Reading/ 
Reasoning 



Logical Reasoning Test 
(LOGREA) 



Reading/ 
Reasoning 



IM Reading Skills Scale 
Reading Scale 1 (READSl) 
Reading Scale 2 (READS2) 



Reading/ 
Reasoning 



Reading Vocabulary Test 
(RVOCTL) 



Anxiety 
Curiosity 



Attitude Toward Course 
Materials 

State Anxiety (STANX) 
State Curiosity (STCUR) 



Anxiety 



Anxiety 



Test Attitude Inventory 
(TAITL) 

Test Worry (TAIWY) 

Test Emotionality (TAIEM) 

Test Anxiety (TAIEX) 

General Attitude Scale 
Trait Anxiety (IRANX) 
Trait Curiosity (TRCUR) 



iledia 

Preference 
Media 

Experience 



General Media Preference 
Scale 

Visual (PREFV) 
Audio (PREFA) 
Print (PREFP) 
Conventional (EXPCI) 
Self-Paced (EXPSP) 



Course Preassessment Testing Battery 

Test Descriptions Author . 

Measures student's mneral ability Hertzka & Guilford 
to judge the logical soundness of (1955) 
meaningful conclusions. 

Measures student's technical infor- McCombs (Note 21) 
mation processing and retention 
skills, under timed conditions, 
on materials extracted from IM and 
MF technical manuals. 



Measures student's comprehension, 
under timed conditions, of terms 
frequently used in air Force 
documents and manuals. 



Deignan (Note 22) 



Measures how t'jnse or apprehensive 
versus interested or motivated a 
student feels about learning the IM 
course materials on an intensity 
dimension. 

Measures, on a frequency dimension, 
student's tendency to foel cognitive 
worry versus emotional versus 
generally anxious when taking per- 
formance or achievement tests. 



Spielberger, 
Gorsuch ^ 
Lushene (1970); 
McCombs-Leherissey 
(Note 23) 

Spielbergor (in pt 
(Note 24) 



ss) 



Measures student's general tendency 
to experience feelings of tension 
and apprehension in situations per- 
ceived as threatening versus it^elings 
of interest in a variety of tecnnical 
areas. 

Measures student's preference for McCoinbs (Note 26) 

visual versus audio versjs printed 

learning materials, as well as his/ 

her experience with conventional 

versus self-paced instructional 

methods 



Spielberger et al . 
(1970); Day (Note 25) 



along with the appropriate variable labels. The results of the IM factor 
analysis are shown in Table 22. As can be seen, the study skills 
variables formed a separate factor following the Varimax rotation pro- 
cfnUw^:^??'' definable factors were derived from the preassessment and 
study skills variables, with the variables from the Study Skills 
Questionnaire loading on the third factor. Of interest for the con- 
struct validity question is that (a) the READCM scale tended to load 
positively on the Reading/Raasoning. Curiosity, and Media Experience 
factors, (b) the MEMORY scale tended to load positively on the Reading/ 
Reasoning factor, (c) the TSTTAK scale tended to load negatively on the 
rZltl^ positively on the Reading/Reasoning factor, (d) the ' 

CONMGJ scale tended to load negatively on the Anxiety factor and posi- 
tively on the Curiosity factor, and (e) the SSQUES total scale tended to 
load negatively on the Anxiety factor and positively on the Reading/ 
Reasoning factor. ^ 

Descriptions of the variables included on the WM preassessment 
battery are given in Table 23. together with the appropriate variable 
labels. The factor analysis results for the WM course, shown in Table 
^4. are similar to those found in the IM course. Seven definable factors 
were derived from the WM preassessment and study skill variable set. 
witn Study Skills Questionnaire variables loading on the second factor. 
Other findings of interest werjb that (a) the READCM scale t^ded to load 
negatively on the Anxiety factior and positively on the Cur.losity factor, 
(b) the MEMORY scale tended td load negatively on the Anxiety factor and, 
positively on the multiple Media Preference factor, (c) the TSTTAK 
scale tended to load negatively on the Anxiety factor, (d) the.CONMGT 
scale tended to load negatively on the multiple Media Preference factor 
and positively on the Curiosity factor, and (e) the SSQUES total scale 
tended to load negatively on the Anxiety factor and on the multiple Media 
Preference factor. 

The differential relationships shown in the IM and WM course factor 
analysis results suggest that, within different trainee populations, 
different patterns of cognitive and affective student characteristic 
variables are related to che students' ratings of their study skills 
in the reading comprehension, memorization, test taking, and concen- 
tration management areas. In addition to the findings of theoretically 
meaningful but differential factor analytic relationships, the data in 
both courses revealed a conceptually distinct study skills factor, in- 
dicating support for the construct validity of the Study Skills Question- 
naire. Also of relevance is the finding that both cognitive and 
affective variables tend to be related to variables in the study skills'--., 
domain, suggesting the need for both cognitive and affective remedial 
strategies in dealing with study skills problems. 

^•4»1.2.2 Predictive Validity . The question of whether the Study 
Skills Questionnaire and its subscales could reliably predict student 
performance in the four AIS courses was examined by a discriminant 
analysis approach. 
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TABLE 22 

Varlmax Rotated Factor Matrix for IM Course Stu4y Skills 
Questionnaire and PreassessmMt Variables (N ? zi3) 



1)7 





FACTOR 1 


VARIABLES 


(Anxiety 


READCN 


-.186 


MEHORY 


-.118 


TSTTAK 


-.469 


CONMGT 


-.303 


ssquES 


-.341 


READSl 


-.068 


READS2 


-.132 


RVOCTL 


-.191 


LOGREA 


-.106 


STANX 


.590 


STCUR 


-.314 


TRANX 


.538 


TRCUR 


-.052 


TAITL 


.961 


TAIWY 


.807 


TAIEN 


.891 


TAIEX 


.816 


PREFA 


.181 


PREFV 


.068 


PREFP 


-.032 


EXPCI 


-.064 


EXPSP 


-.007 



FACTOR 2 
[ Reading/ Reasonlnj 

.210 
.178 
.335 
.001 
.207 
J61 
.592 
.604 
.461 
-.138 
.091 
-.136 
.119 
-.158 
-.161 
-.109 
-.213 
-.003 
.007 
.005 
.213 
.110 



FACTOR 3 
[S tudy Skills: 

.669 
.489 
.480 
.633 
.923 
.082 
.084 
.148 
.141 
-.236 
.140 
»..274 
.036 
-.201 
-.252 
-.153 
-.186 
-.056 
-.058 
.IVl 
.100 
.109 



FACTOR 4 


FACTOR 5 


\neuio 




(Media Experience) 


Prefcrtnce. 


.240 


.224 




-.033 


-.009 




-.101 


.052 




.177 


.099 


1 A1 


.101 


.116 


lUo 


.090 


.125 


• 01 0 


.141 


.079 


-.I/O 


.004 


.106 


• 019 


.033 


.036 


• 091 


-.310 


-.013 




.661 


.062 




-.372 


-.015 


ADC 

• 009 


.649 


.101 


AlO 

• 0J9 


-.058 


-.03f 


AQA 

• 094 


-.043 


-.017 


• Oo*! 


-.052 


-.040 


• 004 


-.126 


-.045 ' 


.159 


.165 


-.072 


.478 


-.089 


.137 


.492 


.322 


.090 


-.487 


.117 


.691 


.058 


.055 


J60 


. -.025 
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TABLE 23: 



Description of WM Course Preassessment Testing Battery 



Factor 



Test and Subscale Names 



Test Descriptions 



Author 



Reading A 
Reasoning 



Reading/ 
Reasoning 



Math Skills 



Math Skills 



Anx»iety 
Curiosity 



Anxiety 



WM Reading Skills Scale 
Reading Scale 1 (READSl) 
Reading Scale 2 (READS2) 



Reading Vocabulary Test 
(RVOCTL) 



Ship Destination Test 
(SHIPDS) 



Math Familiarization Test 
(MATH FT) 

Scale 1 (MATHFl) 
Scale 2 (MATHF2) 



Attitude Toward Course 
Materials 

State Anwety (STANX) 
State Curiosity (STCUR) 



General Attitude Scale 
Trait Anxiety (TRANX) 
Trait Curiosity (TRCUR) 



Measures student's reading compre- 
hension and speed on materials 
extracted from WM technical orders 
and technical manuals. 

Measures a student's comprehension, 
under timed conditions, of terms 
frequently used In Air Force docu- 
ment and manuals. 



McCombs .(Note 21) 



Delgnan (Note 22) 



Measures student's general arithmetic Chris tensen & Guilford 
reasoning or problem solving ability, (1955) 
using specific rules to solve 
problems under timed conditions. 



Measures student's basic math 
skills, under timed conditions, on 
easy and difficult subscales, that 
are required in certain areas of 
the WM course. 

Measures how tense or apprehensive 
versus Interested or motivated a 
student feels about learning the 
WM course materials on an Intensity, 
dimension. 

Measures, on a frequency dimension, 
student's general tendency to ex- 
perience feelings of tension and 
apprehension In situations perceived 
as threatening versus feelings of 
interest in a variety of technical 
areas. 



ATC-Developed 



Spielberger, Gorsuch 
& Lushene (1970); 
McCombs-Leherlssey (Note 23) 



Spielberger et a1 . 
(1970); Day (Note 25) 
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TABLE 23 (Continued) 




Curiosity 



Anxiety 



I 

00 



Media Pref 
erence 



Media Exper- 
ience 



Mechanical Curiosity Scale 
(MECCUR) 



Test Attitude Inventory 
(TAITL) 

Test Worry (TAIWY) 

Test Emotionality (TAIEM) 

Test Anxiety (TAIEX) 

General Media Preference 
Scale 

Visual (PREFV) 
Audio (PREFA) 
Print (PREFP) 
Conventional (EXPCI) 
Self-Paced (EXPSP) 



Measures student's general feel- 
ings of Interest, or tendency to 
become interested in, mechanical 
devices and mechanical principles. 

Measures, on a frequency dimension, 
student's tendency to feel cognitive 
worry versus emotional versus 
generally anxious when taking per- 
formance or achievement tests. 

Measures student's preference for 
visuar versus audio versus printed 
learning modes, as well as his/her 
experience with conventional 
versus self-paced instructional 
methods. 



Unknown 



Spielberger (in press) 
(Note 24) 



KCombs (Note 26) 



Id 




1 n 



TABLE 24 



VarliMx Rotated Factor Matrix for WN Course Study Skjnis 
Questionnaire and Preassessment Variables (N * 2^) 



VARIABLES 


FACTOR 1 
(Anxiety) 


rAtrUR i 

(Study 
Skills) 


FACTOR 3 

(Math 

Skills) 


FACTOR 4 

(Media 

Preference) 


FACTOR 5 
(Curiosity) 


FACTOR 6 
(Readi no/ 
Reasoning) 


FACTOR 7 

(Media 

Experience) 


READCM 


-.202 


.666 


.063 


-.121 


.238 


.050 


.055 


MEMORY 


-.20u 


.611 


.023 


.218 


-.010 


.090 


-.039 

• ww^ 


TSTTAK 


-.488 


.540 


.093 


-.023 


.060 


• 095 


064 

• wW~ 


CONMGT 


-.167 


.590 


.087 


-.456 


.215 


-.069 

9 WW ^ 


.056 

• wwV 


SSQUES 


-.303 


.927 


.088 


-j^206 


.171 


.032 

• Www 


.047 

• w^ r 


REA0S1 


-.028 


.073 


.104 


-7o75^ 


.015 


.455 


.072 

• W • Bp 


READS2 


-.062 


-.049 


.093 


-.00l\ 


.075 


.510 


.088 

• www 


RVOCTL 


-.216 


.127 


.173 


-.061 ' 


-.029 


.511 


.099 

• W«r «r 


SHIPDS 


-.143 


.051 


.330 


.070 


-.020 


.358 


.047 

• w • r 


STANX 


.563 


-.251 


-.134 


.039 


-.398 


-.048 


-. i54 

• ■ w^ 


STCUR 


-.250 


.215 


.016 


-.003 


.647 


.166 


.064 

• W W w 


MECCUR 


-.009 


.124 


-.017 


.049 


.692 


-.054 


.197 


PREFV 


.137 


-.036 


-.002 


.482 


.154 


-.052 


.044 

• W f T 


PREFA 


J46 


.147 


-.047 


.462 


.034 


-.319 


.040 

• w~w 


PREFP 


-.063 


.124 


-.023 


-.487 


.094 


•f.036 


-.073 


EXPCI 


.008 


.061 


-.007 


.066 


.146 


.159 


.679 


EXPSP 


-.061 


.002 


.076 


.080 


.100 


.115 


.685 

o www 


MATHFl 


.036 


.059 


.511 


.035 


-.003 


.140 


-.049 

• w » ^ 


MATHF2 


-.150 


.047 


.881 


-.073 


.025 


.149 


.109 

• ■ W«r 


MATHFT 


-.109 


.058 


.995 


-.044 


.025 


.164 

• 1 w~ 


061 

• vw 1 


TAIWY 


.829 


-.232 


-.054 


.156 


-.096 


-.143 


-.008 


TAIEM 


.J|95 


-.164 


-.008 


.135 


-.031 


-.123 


.051 


TAIEX 


.829 


-.217 


-.128 


.127 


-.048 


-.132 


-.069 


TAITL 


.959 


-.205 


-.052 


.150 


-.059 


-.136 


.003 



The purpose of these analyses was to determine if the various study 
skills variables could reliably distinguish students in the least 
efficient and least effective quartiles, on the block and lesson com- 
pletion time and score criterion variables, from those remaining 75 
percent of the students who were having less difficulty completing the 
course quickly and successfully. These analyses were restricted to the 
IM and WM courses since these courses had ,the largest numbers of stu- 
dent samples available on the Study Skills Questionnaire administered 
at the end of the first block. 

In the IM course, block level data considered appropriate for dis- 
criminant analyses were block completion times on Blocks 2 through 5, 
block test failures on an early (Block 2) and a late (Block b) block, and 
block test scores on these same two blocks. In addition, cumulative 
lesson completion times and average lesson test scores were examined for 
these two blocks. 

Results of the discriminant analyses on the IM block level data are 
reported in Table 25. The rtcults indicated that the study skills 
variables were moderately effective in discriminating slow from fast 
students, correctly classifying between 61.3 and 69.8 percent of the 
students, with a slight gain in predictability ^-om the early to later 
blocks. When completion times across the four blocks were summed, the 
scales were quite effective in reliably discriminating the slowest 25 per 
cent of the students from the remaining students, correctly classifying 
67.3 percent of the students. 

With respect to number of block test failures, the scales were 
moderately effective in discriminating students with one or more block 
test failures from those who passed the block tests on the first attempt. 
Ill this case, predictability actually improved somewhat from the second 
(5a. 2 percent correctly classified) to the fifth (61.3 percent correctly 
classified) block. 

Finally, it was considered of interest to determine tha scales' 
power to discriminate in the bottom quarter of the block test score dis- 
tribution from the remaining 75 percent. Again, the scales were found 
to be quite effective in discriminating between these two groups and, 
again, predictability was found to improve slightly fror.; the second to 
the fifth block (63.0 versus 64.3 percent correctly classified). When 
Block 2 through 5 scores were summed and the lowest 25 percent dis- 
criminated from the remaining 75 percent, 65.1 percent of the students 
were correctly classified. 

In general the MEMORY, SSQUES, and CONMGT scales appeared to be 
most effective in predicting block times, while tne SSQUES, MEMORY, and 
TSTTAK scales were most effective in predicting block failures and test 
scores. 

Results of the lesson level discriminant analyses on the early 
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TABLE 25 



IM Course Discriminant Analysis Results for Block Level Data 



Criterion 


Bottom 25X 


Remalnir 






% Correctly 


Order of 


Variables 


Cutoff 


H 


Cutoff 




Classified 


Predictors 


Block 2 
Time (In 
minutes) 

Block 3 
Time (in 
minutes) 


51877 
^1678 


67 

72 


< 1877 

< 1678 


202 

\ 
2201 

/ 


36.4** 
j24.2** 


68.4X 
64.4X 


MEMORY 

ssquEs 

READCM 
CONMGT 
TSHAK 

MEMOR? 
SSQUES 

REAdCM 
TSUAK 


Block 4 
Time (In 

tin mi^A^ 1 


^1154 


71 


<1154 


211 


14.5** 


61. 3X 


CONMGT 
MEMORY 

nf* A n#*aa 

READCM 
TSHAK 


Block 5 
Time (in 
minutes) 


^1733 


74 


<1733 


221 


4e.4** 


69. 8X 


SSQUES 
MEMORY 
READCM 
CONMGT 
TSHAK 


Block 2-5 
Time (in 
minutes) 


^ 6545 


61 


<6545 


181 


30.6** 


67. 8X 


MEMORY 
SSQUES 
CONMGT 
READCM 
YSHAK 


Block 2 
Failures 


^1 


45 


0 


168 


5.8* 


58. 2X 


CONMGT 

TS;. il 
SSQUES 


Block 3 
Failures 





70 


0 


158 


12.8** 


61. 8X 


MEMORY 
TSTTAK 
SSQUES 


Block 2 
Score 


<73 


79 


^73 


221 


20.3** 


63.0X 


SSQUES 
MEMORY 
READCM 
CONMGT 
TSHAK 


Block 5 
Score 


<69 


60 


^69 


240 


24.7** 


64. 3X 


MEMORY 
SSQUES 
TSTTAK 

READCM 


Block 2-5 
Score 


< 296 


74 


^296 


224 


27.2** 


65.U 


TSHAK 
SSQUES 
MEMORY 
CONMGT 
READCM 



* p < .01 ** p < .001 
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(Block 2) and late (Block 5) blocks of the IM course are shown in Table 
26. Again, it should be noted that the analyses examined the performance 
of the bottom 25 percent of the students from the remaining 75 percent 
with respect to lesson completion times and scores. The cutoff values 
shown refer to cumulative lesson completion times and average first 
attempt lesson test scores for all of the lessons within each of the 
blocks. The study skills scales were found to be moderately effective 
in reliably discriminating between these cwo groups on both the time 
and score dimensions, with the percent of students correctly classified 
ranging from 61.2 to 66.1 pe.cent. It is important to note that there 
was no great loss (apparently, in fact, a slight gain for the time 
variable) in predictability from the early to the later block. The 
TTSTAK and SSQUES total scales were most effective in discriminating 
between groups on the time dimension, while TSTTAK and CONMGT scales 
apfieared to do the best job in discriminating between groups with re- 
spect to lesson scores. 

In the Wli course, which consisted of 14 blocks, the number of cases 
available in the later blocks was consistently less than 100 per block, 
which v^ould imply large instability in parameter estimation. Therefore, 
the blovk level data considered appropriate for analysis in the WM 
course consisted of block completion times for Blocks 2 through 8, block 
test failures on an early (Block 2) and a relatively late (Block 8) block, 
and block test scores on these same two blocks. Cumulative lesson com- 
pletion times and average lesson test scores were also examined for 
these two blocks. 

Results of the discriminant analyses for the WM block level data 
are reported in Table 27. The same procedures were followed as for the 
IM course analyses. The results indicated that the study skills var- 
iables were moderately effective in discriminating inefficient (lowest 
25 percent) from efficient (remaining 75 percent) students with respect 
to individual block completion times. Excluding Blocks 3 and 7, (in 
which the chi-square values were not significant at the p< .05 level), 
between 61.0 and 75.7 percent of the students were correctly classified. 
It is likely that the classification failure in Block 7 could be attri- 
buted to the fact that almost all of the instruction in this block was 
presented via audio-visual media and thus was largely device-paced rather 
than being as dependent on student efficiency. When completion times 
were sunined across the seven blocks, the scales were quite effective in 
reliably discriminating the slowest 25 percent of the students from the 
remaining 75 percent, correctly classifying 68.9 percent of the students. 

In Blocks 2 and 8, the scales were highly effective in discriminat- 
ing students with one or more block test failures from those who passed 
the block tests on the first attempt. As had been the case for the IM 
course, predictability actually increased from the earlier (Block 2, 
68.3 percert correctly classified) to the later (Block 8, 71.? percent 
correctly classified) blocks. 
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TABLE 26 

IM Course Discriminant Analysis Results for Lesson Level Data 



Criterion 


Bottom 25% F 


temaining 75% 




'% Correctly 


Order of 


Variables 


Cutoff 


N 


Cutoff 


N 


Classified 


Predictors 


Block L 
















Lesson 
Time 


> 1732 


55 


«1732 


164 


11.0* 


6]. 2% 


TSTTAK 
SSQUES 
MEMORY 
CONMGT 


Lesson 
Score 


< 579 


55 


>579 


172 


18.6** 


64.3% 


TSTTAK 
SSQUES 
CONMGT 
READCM 
MEMORY 


Block 5 
















Lesson 
Time 


> 1416 


71 


<1416 


209 


28.9** 


66.1% 


SSQUES 
MEMORY 
READCM 
CONMGT 
TSTTAK 


Lesson 
Score 


< 606 


70 


>606 


217 


18.6** 


62.7% 


CONMGT 
SSQUES 
MEMORY 
TSTTAK 
.xcADCM 



* p < .01. 
** p < .001. 
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TABLE 27 



UM Course Discriminant Analysis Results for Block Level Data 



Criterion 


Bottom 25X 


Remaining 


75% 


w2 


% Correctly 


Order of 


Variables 


Cutoff 


N 


' Cutoff 


N 


X 


Classified 


Predictors 


Block 2 
Time (in 
minutes) 


^ 451 


38 


<451 


106 


38.0*** 


75. 7X 


SSQUES 
CONMGT 
READCN 
TSHAK 


PHock 3 
Time (In 
minutes) 


> 583 


38 


< 583 


108 


1.8 


55. 5X 




Block 4 
Time (In 
minutes) 


:^ 428 


36 


<428 


107 


5.1* 


59. 4X 


SSQUES 
MEMORY 
READCN 
CONMGT 
TSTTAK 


Block 5 
Time (In 
minutes) 


^ 1603 


34 


< 1603 


102 


6.6** 


61.0% 


SSQUES 
TSHAK 
READCM 
CONMGT 
MEMORY 


Block 6 
Time (In 
minuces) 


>1448 


32 


< 1448 


94 


6.2* 


61. U 


SSQUES 
TSTTAK 
MEMORY 

Cf\ktkif*T 

LUNMuT 
READCH 


Block 7 
Time (In 
minutes) 


>624 


35 


< 624 


106 


2.1 


56.0% 




Block 8 
Time (In 
minutes) 


>1569 


35 


<1569 


104 


18.7** 


68.3% 


SSQUES 
TSTTAK 
READCM 
MEMORY 
CONMGT 


Block 2-8 
Time (in 
minutes) 


> 6496 


13 


^ 6496 


61 


10.6** 


68.9% 


READCM 
SSQUES 
MEMORY 
TSTTAK 


Block 2 
Failures 


>1 


i'58 


0 

.•4' 


13 


36.2*** 


68.3% 


MEMORY 
TSTTAK 
CONMGT 
READCM 


block b 
Failures 




129 


0 


10 


26. b*** 


71 : 9% 


READCM 
TSTTAK 
SSQUES 
CONMGT 
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TABLE 27 (Continued) 



Crl terion 


Bottom 25% 


Remaining 75% 


X2 


% Correctly 


Order of 


Variables 


Cutoff 


N 


Cutoff 


N 


Classified 


Predictors 


Block 2 


< 72 


21 


7 72 

• 


125 


2.7 


56.8% 


REAOCM 
TSTTAK 
MEMORY 
CONMGT 
SSQUES 


Block 8 
Score 


< 71 


36 


^ 71 


110 


5.4* 


59.6% 


TSTTAK 
SSQUES 
CONMGT 
READCM 


Biock 2-8 
Score 


< 545 


13 


:^ 545 


62 


1.2** 


69.3% 


REAOCM 
MEMORY 
CONMGT 
SSQUES 



* p < .05. 
** p < .01. 
*** p < .001. 



When Block 2 



The four study skills scales were only fairly effective in discrim- 
inating the lowest 25 percent of the students from the remaining 75 per- 
cent with respect to the block test score distribution, with 56.8 percent 
of the students classified in Block 2 (chi-square not significant at 
p < .05), and 59.6 percisnt correctly classified in Block 8. 
through 3 scores were summed, however, the discrimination results were 
much improved, with 69.3 percent of the students correctly classified. 

In general, the SSQUES total scale was most effective in predicting 
block completion times, while the REAOCM and TSTTAK scales were most 
effective in predicting block test failures and scores. 

Results of the lesson level discriminant analyses on the early 
(Block 2) and late (Block 8) portions of the WM course are presented in 
Table 28. As was the case for the IM lesson level analyses, the two 
student categories defined on each (time and score) dimension were those 
considered to be displaying unsatisfactory (bottom 25 percent) or satis- 
factory ^remaining 75 percent) performance. Again, the cutoff values 
shown pertain to cumulative lesson completion times across all lessons 
in the J lock and average first attempt lesson test scores for all- 
lessons having cognitive criterion tests. The study skills scales were 
moderately effective in discriminating between the two groups In terms of 
time (61.0 and 68.2 percent correctly classified), but were less effective 
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TABLE 28 

WM Course Discriminant Analysis Results for Lesson Lt ' Data 



Criterion 


Bottom 2b% 


Remaini 


no 75% 


^2 


% Correctly 


Order of 


Variables 


Cutoff 


N 


Cutoff 


N 


Classified 


Predictor 


Block 2 
















Lesson 
Time 


> 422 


68 


< 422 


199 


3.0* 


61.0% 


SSQUES 
CONMGT 
READCM 
MEMORY 
TSTTAK 


Lesson 
Score 


< 151 


59 


^ 151 


222 


2.6 


54.8% 


SSQUES 
MEMPRY 
CONMGT 

TSTTAK 
READCM 


Block 8 
















Lesson 
Time 


> 602 


28 


< 602 


82 


14.6* 


68.2% 


SSQUES 

TCTTAl' 

READCM 
CONMGT 


Lesson 
Score 


< 307 


36 


> 307 


109 


3.7 


57.9% 


TSTTAK 
READCM 
CONMGT 
MFMORY 
SSQUES 



* p < .01. 



in discriminating between groups on the score dimension (54.8 and 57.9 
percent correctly classified). Once again, there is a consistent 
pattern of increasing predictability from the earlier to the later 
block. The SSQUES total and TSTTAK scales appeared to be the most 
effective in discriminating students categorized into the Low Group from 
those in the High Group, with SSQUES being more effective on the time 
dimension and TSTTAK, on the score dimension. 

Finally, of interest in assessing the predictive validity of the 
Stidy Skills Questionnaire and its subscales was the question of the 
relative power of these variables to discriminate the poorest 2b percent 
of students on the lesson and block time and score dimensions from the 
remaining 75 percent of the students, as compared with the standard set 
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of preassessment predictors utilized in the JM and WM courses. To answer 
this question, the same set of discriminant analyses were calculated on 
the IM and WM course dat? as had been calculated with only the Study 
Skills Questionnaire variables. For the IM course, the preassessment 
predictor set consisted of 24 course-specific cognitive, affective, and 
background information predictors; for the WM course, tha preassessment 
predictor set consisted of 26 course-speciyic predictors. 

The results of the IM course analyses generally indicated that the 
preassessment set correctly cl-'ssified approximately (a) 3 percent more 
students on the block time criterion, (b) 9 percent more students on the 
block score criterion, (c) an average of five percent mor students on 
the lesson time criterion, and (d) an average of 4 1/2 percent more 
students on the lesson score criterion, as compared to the Questionnaire 
analyses. The WM course analyses yielded similar results, indicating 
that the preassessment set correctly classified approximately (a) 21 per- 
cent more students on the block time cr- terion, (b) 11 percent more 
students on the block score criterion, (c) an average of 5 percent more 
students on the lesson time criterion, and (d) an average of 15 percent 
more students on the lesson score criterion, as compared to the Question- 
naire analyses. Thus, the larger preassessment sets were able to 
correctly classify a larger proportion of the students^^J^ticularly at 
the block level in the WM course. An additional set of discriminant 
analyses, however, which utilized variables from both the Questionnaire 
and preassessment sets indicated that for both the IM and WM courses, 
all five of the Study Skills Questionnaire variables were in the set of 
the most significant predictors of group membership for the lesson and 
block level analyses (as defined by changes in Raos V discrimination 
index). These results imply that the Questionnaire variables are measur- 
ing additional factors important to t^e prediction of student performance 
in a CMI environment, which was also substantiated in the factor analysis 
results. 

In sunwnary, these results indicate that the Study Ski lis Question- 
naire and its subscales demonstrated i'good preliminary construct validity 
in both the IM and WM courses, and promising predictive validity. Wi':h 
respect to predictive validity, it is particularly important to note 
that the degree to which .i.^ scales correlated with subsequent student 
performance did not decrease as a function of time in the course and, in 
several cases, actually improved. It is also of interest that different 
patterns of predictor relationships were found for differing criterion 
variables, suggesting that the study skills variables are sensitive pre- 
dictors of both different kinds of CMI criteria (time, score) and 
different kinds of course content (IM, WM blocks or lessons). In 
addition, although the smaller set of Study Skills Questionnaire 
variables was not as effective as the larger preassessment variable 
sets in correctly classifying unsatisfactorily versus satisfactorily 
performing student groups, the disciminant results were encouraging for 
those CMI environments which do not accorfmodate preassessment testing. 
These validity results and conclusions are, of course, tentative until 
substantiated in cross-validation studies. 
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4.0 DISCUSSION AND CONCLUSIONS 



4.1 Discussion of Findings 

The goals of this project were to (a) determine the characteristic 
problems which students encounter in a computer-managed instructional 
system and those strategies which effectively help students cope with or 
adapt to these problems, (b) design, develop, implement, and evaluate 
a small set of self-contained instructional modules fur increasing the 
effectiveness with which stud» its adapt to and perform in a CMI environ- 
ment, and (c) investigate procedures for individualizing the assignment 
of these modules so as to minimize total completion times and training 
costs. The skill assessment and training materials developed in 
response to these goals wtre an Orientation to CMI module, a Time Manage- 
ment module, four Study Skills modules (Reading Comprehension, Memori- 
zation, Test Taking, Concentration Management), and a Study Skills 
Questionnaire. In addition, an Instructor Orientation and Training 
package was developed in which a workshop format was used to train 
instructors in diagnostic and tutorial skills necessary for f|ffective 
student use of the study skills materials. The following sections 
summarize the results of the operational and summative evaluations of 
these skill training materials, followed by a discussion of evaluation 
conclusions and recommendations for future research. 

4.1.1 Student Skill Training Modules . As will be recalled, the 
Orientation module was subjected to an extensive formative evaluation 
with both students and instructors in the AIS environment, and was 
administered to a number of students in the PME course in an Orientation/ 
No Orientation design operational tryout. Due to small student flow 
during this later evaluation period, no reliable differences were 
detected between attitudes toward CMI or block performance for those 
students who received and those who did not receive the Orientation 
module. The average time which students spent to complete the module 
was, however, only 42 minutes, suggesting that the time for such an 
orientation is not excessive and that subsequent evaluations of the 
module could be expected to reveal student performance or attitude 
benefits which are well worth this minor investment of student time. 

Following formative evaluation and revision, the effectiveness of 
the Time Management module was assessed in two phases. The question of 
interest in Phase I was whether the combination of Time Management skill 
training and progress targeting and feedback vStudent Progress M-nagement 
Component) would result in significant reductions in block and course 
completion times. 

There is little doubt that the combination of the Time Management 
module and Student Progress Management Component did result in substan- 
tive time savings— an (1.2 percent reduction in the time required for 
students to complete the six blocks of the IM coursf.. This 11.2 percent 
savings represented 2.87 fewer days spent in a learning center by the 
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average .student. Since the normal IM course entry rate is 60 students 
per week for 50 training weeks per year, the total yearly savings 
anK)unt$ to 8,600 student training days. 

The Phase I block score, block failure rate, and student attitude 
results demonstrated less positive, but not totally unanticipated, 
effects. Tha*. is, block test $cores were lower, failure rates Were 
higher, and altitudes tended to be Jess positive for students in the 
progress management condition as cJPared to the no-management condition. 
These results are consistent with the findings of Johnson et al. (1972) 
with respect to attitudes and with those of Colton (1974| for criterion 
test scores, both of whom investigated the effects of giving students 
predicted completion time information. The results also agree with 
Fernald (1975) and Reiser and Sullivan (1977), who investigated the 
effects of instructor-paced versus student-paced conditions. Increased 
emphasis on completing the course quickly appears to have caused students 
to accept a greater risk of failure on end-of-block criterion tests, and 
the lower attitude scores may reflect feelings of conflict about this 
trade-off of shorter times for lower scores. 

• 

A study by l^cMillan (1977) suggests another explanation for the less 
positive attitudes of students in the management condition. The inter- 
active effects of students' degree of effort and the nature of written 
instructor feedback on student attitudes toward the subject were inves- 
tigated in four univer<;ity classes. High and low effort assignments 
were studied within conditions of high praise or no praise. McMillan 
found that students in the high effort/high praise condition formed 
significantly more positive attitudes than did students in the other 
three groups. The poorest attitudes were found for students in the high 
effort/no-praisv* condition. These findings imply that the progress 
management condition (a high effort condition relative to no-management) 
students may not have received enough instructor praise or encouragement 
for their efforts in maintaining their target rates to provide the 
environment required for positive attitude formation. 

The question of interest on Phase II of the evaluation was whether 
the Time Management module per se, in the presence of the Student 
Progress Management Component, contributed significantly to training 
time reductions. The data from this second phase indicate a substantial 
time savings attributable to the Time Management module— 14.77 percent 
for students v/ho completed all six blocks of the IM course. A word of 
caution is in order, however, in interpreting these time savings. The 
samples in the later blocks were small and were comprised of the faster 
students. Even though no interactions were found between the effect of 
the module and predicted course completion time, the data from the later 
blocks may not be representative. Disregarding the first block, which 
is confounded by the presence of the module itself, the comparisons made 
in the second and third blocks, both based on reasonably large ri,'s, 
probebiy provide the most reliable estimate of time savings attributable 
to the module— on the order of 9 percent. 
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The finding that the Time Management module led to improved perfor- 
mance relative to a no-module group is congruent wi th several studies 
reviewed in Section 1.2.2 of this report. Croft et al. (1976) found 
that PSI students in a variety of monitoring conditions took fewer • 
sessions to complete the course than did students who merely paced 
themselves. Similar studies by Yates and Zimbardo (1977) and r^ers 
(1978) found performance improvements for student groups who were taught 
self-monitoring techniques in a self-paced environment and were required 
to use these techniques throughout the course, as comp!^red to no- 
monitoring student groups. Thus, these studies suggest that procedures 
which require students to observe, record, or graph their progress are 
related to improved performance in self-paced learning environments. 

It is of interest to assess the time savings which can be attributed 
to the basic Protjress Management Component, "to the module, and to a third 
factor— the continued presence of Progress Management over time. Table 29 
summarizes several comparisons based on cumulative times to complete 
Blocks 2 and 3 by Phase I and II evaluation and control group students. 

As Table 29 indicates, introduction of the combination of the Manage- 
ment Component and Time Management module resulted in a 12.2 percent 
reduction in these blocks as measured during the Phase I evaluation. 
There is no available measure of the immediate effect of the Management 
Component by itself. The module, in the presence of the Management 
Component, contributed a 9.1 percent time reduction as measured during 
Phase II. Approximately 4 months elapsed between the end of Phase I and 
the beginning of Phase II, and comparison of Phase I and II evaluation 
groups indicates that the combined effect of Management Component plus 
module resulted in an additional 4.27 nour reduction (7.1 percent) in 
these blocks during this period. Thus, the total effect of the full 
Management Component (including the module), after a period of time, was 
an la.b percent reduction in completion times for these two blocks. The 
final available comparison implies that the Management Component, by 
itself, would have resulted in a 10.3 percent savings, given time to 
have its full effect. This may be an overestimate, however, since both 
the Management Component and module were in effect during the intervening 
period. 

Two major implications can be drav^n from these results. First, both 
parts of the total Progress Management Component contribute to improve- 
ments in student efficiency which are roughly equivalent (i.e., 9.1 
percent for the Module, 10.3 percent for the Progress Management Com- 
ponent). 'Second, the effects of the total Progress Management Component 
continued to increase over time. 

The phenomenon of improving performance over time could be 
attributed to several factors. First, initial negative reactions by 
students and instructors to Progress Management may have diminished as 
its effectiveness ^or improving trainee efficiency was demonstrated. 
Some support for this hypothesis can be found in the more positive Phase 
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TABLE 29 



TIME SAVINGS IN BLOCKS 2 AND 3 
ATTRIBUTABLE TO VARIOUS TREATMENTS 



Treatment 


Comparison Groups 


Time Difference 
in Hours 


^ Percentage 
Time Savings 


Management Component 
plus Module 


Phase I Control - 
Phase I Evaluation 


8.38 


12.2 


Management Com- 
ponent Only 


Not Available 






Module Only 


Phase II Control - 
Phase II Evaluation 


5.61 


9.1 


Time: Given Management 
Component plus Module 


Phase I Evaluation - 
Phase II Evaluation 


4.27 


7.1 


Time plus Management 
Component plus Module 


Phase I Control - 
Phase II Evaluation 


12.65 


18.5 


Tinie plus Manage- 
ment Component 


Phase I Control - 
Phase II Control 


7.04 


10.3 


Time plus Module 


Not Available 







II student attitude data as compared with the Phase I attitude data. 
Second, the additional 4 months of Management Component operation may 
have resulted in greater expertise on the part of the instructors in 
the use of the Progress Management Component. 

Score and block failure rate data also support the contention that 
initial benefits increased over time. Phase II evaluation group students 
were generally found to score as well and experience the same number of 
block failures as students without the module. Further, this level of 
achievement was approximately that found for students prior to implemen- 
tation of Progress Management. These findings suggest that the poor 
test performance effects observed when Progress Management was first 
implemented were only temporary. 

That perceptions of Progress Management improved as experience with 
it increased is substantiated to an extent by the Phase II student 
attitude data. Attitude items which were most negative in Phase I tended 
to become more positive in Phase II, particularly those items dealing 
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with instructor help and encouragement. These findings agree with those 
of McMillan (1977) in that more positive attitudes would be expected in 
this high effort/high (or moderate) praise condition. 

A final point concerns the time required to complete the Time 
Management module relative to the savings attributable to this training. 
In Phase II, evaluation group students had recovered the time spent on 
the module by the end of the first block. This result is quite consis- 
tent with Anderson's (1976) conclusions that early skill training "costs" 
are amortized quickly when students begin using these skills to improve 
their performance. 

In the Study Skills training area, the findings of relevance are 
those found during the operational tryout of the four modules in the IM, 
MF, and WM courses. Although the data from this evaluation are limited 
with respect to number of samples, the findings of dramatic student 
improvement in block times and scores following study skills remediation 
are so consistent that it appears fairly clear that this training met 
the goal of increasing student efficiency and effectiveness in a CMI 
training environment. There was also a suggestion that providing 
students with study skills training improved their perceptions of their 
study skills, particularly if training was given in more than one study 
skills area. These findings are consistent with the results of a number 
of studies reviewed in Section 1.2.3 of this report (e.g., Groveman et. 
al, 1977; McReynolds & Church, 1973). 

Two major implications of the study skills training results are that 
this relatively short training (a) continues to show its effect in im- 
proved student performance throughout the course, and (b) tends to hav" 
its most dramatic effect on the performance of students who were assign- 
ed all four of the Study Skills modules. The first of these phenomena 
would be expected on the basis of the strategies used to modify student 
study behaviors, i.e., active information processing strategies and 
cognitive self-control strategies. Since emphasis is placed on active 
and meaningful information processing, the continuing practice itself, 
and its inmediate resultant reinforcement, would tend to maintain and 
strengthen the desired behaviors over time. Similarly, it would be 
expected that the cognitive self-<:ontrol strategies, such as the positive 
self-talk scenarios used in the Concentration Management and Test Taking 
modules, would become habitual with practice, 

' - That students assigned all four Study Skills modules tended to im- 
prove their performance to a greater extent than students receiving only 
one module (particularly if the one module was Reading Comprehension) 
may well have been due to the sheer mass of the study skills training, 
but it could also be attributable to inappropriate instructor matcnes 
of Teaming materials with student needs or the characteristics of the 
particular students selected for study skills remediation. It may be 
that the students selected had other aptitude or motivational character- 
istics that differentially moderated the effects of the study skills 



training. This suggests that an individualized assignment procedure 
which uses student characteristic data in assisting instructors' diagnoses 
would further improve the effectiveness of study skills training. 

4.1.2 Study Skills Questionnaire . The results relevant to the 
use of the Study Skills Questionnaire as a diagnostic tool for helping 
to Identify those s'-.udents in need of study skills remediation were 
quite promising. Not only were the questionnaire and its subscales 
found to demonstrate good reliability and construct validity, but the 
results of the predictive validity analyses supported the pov«r of this 
measure to discriir.inate those students who would perform satisfactorily 
versus poorly in i\ CMI environment such as the MS. The Quesionnnaire 
reliably discriminated between the groups for biock times and scores, 
as well as lesson times and scores, suggesting the sensitivity of the 
questionnaire to different CMI criterion variables. A finding of 
importance was that the Questionnaire's ability to predict performance 
generally did .,ot decrease from early to late blocks of both the IM and 
UM courses. 

A recent study by Centra (1977) supports the^relationship found 
between student study skills ratings and subsequent course performance. 
Centra investigated whether students' ratings of their skills in seven 
different undergraduate courses (chemistry, biology, introdi^rtory psych- 
ology, three math courses, physics) would be related to end-of-course 
exam performance. Ratings of more global abilities were most highly 
related to student performance, whereas more specific ratings of the 
content, difficulty, or procedures of the courses were not. This 
suggests that student ratings of their study skills are most predictive 
when they are asked for general or global opinions of their abilities 
in certain general skills required for the course— a characteristic of 
the items of the Study Skills Questionnaire. 

The resul s of a study by Bomstein, Hamilton, Miller, Quevillon, 
and Spitzform (1977) further suggest that not only are self-rating scales 
more likely to be related to student performance than ability measures, 
but that there are a number of techniques that can be used to improve 
the reliability and validity of self-report scales. For example, 
Bomstein et al. report that taking the time to tell students that they 
are independent thinkers, that they are believed to have high integrity 
and to be able to evaluate themselves honestly, and that inaccurate 
reporting of data would result in loss of time, money, and energy has^^ . 
the payoff of increasing the fidelity of self-reports. Thus, these 
findings suggest that the reliability and validity of the Study Skills 
Questionnaire could be further enhanced by similar procedures. 

Although the preassessment variable sets available in the AIS 
environment were somewhat more effective than the Questionnaire variables 
in discriminating those students who perform satisfactorily versus poorly 
with respect to completion times and criterion test scores, there are a 
number of issues related to the operational application of the Study 
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Skills Questionnaire in a CMI environment that should be addressed. 
First, for those CMI environments which do not support preassessmcnt 
testing or the use of precourse student data in perfoiiria nee predictions, 
it may be more efficient to simply implement the Questionnaire for pre- 
dictive and diagnostic purposes, rather than designing and implementing 
some type of precourse assessment procedure. The second issue is 
related to the intended use of the Study Skills Questionnaire in a CMI 
environment as a prescriptive and/or diagnostic tool. On the basis or . 
the present findings, it would appear that the Questionnaire could at 
least be used to supplement the prediction of student performance, in 
that it serves a highly useful function in the diagnosis of Pa;;tjcular 
weaknesses ^n student study skills, and thus facilitates the instructor s 
remediation decisions. 

A final issue concerns the use of the Questionnaire in the diagnosis 
of particular study skill weaknesses, and whether it might more 
efficient to simpij administer all four Study Skills Modules Jo students 
predicted to have trouble satisfactorily perfomiing in a CMI course, 
rather than using the Questionnaire to select particular Study Skills 
Modules for particular student needs. Related to this issue is the 
question of how important it may be to give the instructors a ""ore 
Expanded role in using the Questionnaire data to make fined diagnostic 
I^d remedial decisions. All of these issues. ^^^^^^ " 
siderations in arriving at reconmendations regarding how the Study Skills 
Questionnaire should be used in various CMI environments. 

4.1.3 Instructor Orientation and Training . As was discussed pre-- 
viously in this report, in the process of designing fd/^Pien^nting the 
student skills training modules, it became apparent that additiona 
mechanisms would be necessary to effectively transition these modules 
TtS'trCMr^train n'enviroLnt. That is. in/^-f^^.^^f-/,^: as 
and anoreciation of the goals and objectives of this skills training, as 
; VSrrhr?:°p7em;niati'on procedures and ^^^Jj^^f'^irs lm- training' 
sidered to be critical to the potential success of such skills training. 
For this reason, it was decided that an instructor orientation and train- 
ina packaSethoJld be developed to acquaint instructors with project 
To is a d'with basic diagnostic and tutorial skills required or the 
individualized assignment of the student study skills materials This 
instructor package and its associated procedures was developed in place 
if ^i;^ originally planned computer-based individualization procedures. 

Data for evaluation of the Instructor Orientation Training 
package were provided both by the instructor critiques of the workshop 
trang content and procedures, and instructors' subsequen use of 
their new skills in assigning students to particular study skills 
materials With respect to the critique 'data, instructor comments from 
The fiJmaiive and operational tryouts of the workshops were sen^'-ally 
fa'^vora'e with a majority of the 27 Participating ^nstructors in eating 
that, thev liked both the content and format of the workshops, mese 
fSnSingl. then. sSpport those of previous researchers who have reported 
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on the success of a workshop format for instructor training (e.g.. 
Cadenhead, 1976; Gall et al., 1978; Rash & Grinm, 1976). ^ ' * 

An index of the success of the instructor orientation and training 
procedures is also provided by examination of the number of students per 
course who were subsequently assigned some type of study skills remedi- 
ation, and their performance following this remediation. Of the 27 in- 
structors who participated in the formative pr operational tryouts of the 
workshops, 18 had subsequent changes in their work assignments (e.g.. to 
curriculum writing), leaving only nine instructors (three from IM, one 

1 ^rom PME) who were in a position to assign the Study Skills 

modules during the 10-week evaluation period. Referring to Table 9 in 
Section 3.3.1, the three IM instructors assigned materials to seven 
students, for an average of about 2.3 assignments per instructor. The 
cne MF instructor assigned materials to two students. The five WM in- 
structors assigned materials to 12 student?, for an average of about 2.4 
assignments per instructor. As was pointed out in Section 3.3.1, how- 
ever, these numbers are misleading in that discussions with instructors 
and data clerks in these courses indicated that substantially more 
materials were assigned to students but were not recorded in the AIS 
data base. Thus, the Instructor Orientation and Training was at least 
moderately successful in promoting the remediation of student study 
skills problems— a finding further substantiatecf by the consistent im- 
provements in student block times and scores following this remediation. 

Another indication that the Instructor Training/Study Skills 
package, as a whole, was successful is reflected in the ancedotal data 
obtained from informal conversations with individual instructors. With 
few exceptions, the instructors who participated in the program indicated 
that although they were initially skeptical, their experience with the 
materials convinced them that students assigned the materials generally 
show dramatic improvement, in course performance. Many of them, more- 
over, felt that the Study Skills Questionnaire and four modules should 
be assigned to all students at the beginning of their course. Such a 
decision must be made by the management of each course, but it is 
encouraging to note that the instructors who used the study skills 
materials gained confidence in their ability to teach students new, more 
efficient and effective behaviors. These anecdotal data support the con- 
tention that an Instructor Orientation and Training package can posi- 
tively affect instructor perceptions and attitudes about their CMI roles. 

4.2 Evaluation Conclusions 

The conclusions from this project are discussed in the following 
sections, with respect to the student skills training modules, the Study 
Skills Questionnaire, and instructor orientation and training, 
respectively. 

4.2.1 Student Skills Training Modules . On the basis of the results 
of the formative and summative evaluation activities complete I on the 
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Orientation modul'3. Time Management module, and four Study Skills 
modules, the following general conclusions can be drawn: 

1. Providing students with an orientation to the novel aspects of 
a CMI environment is both feasible and efficient. The effectiveness of 
'such an orientation remains to be demonstrated with respect to improving 

student training efficiency and effectiveness, or improving their 
attitudes toward the new learning processes and instructional environ- 
ment. 

2. Providing students with rudimentary time management skills and 
some method for maintaining these skills throughout the course is highly 
feasible and cost-effective, particularly when this skills training is 
combined with a system for predicting completion times and providing^ 
rate of progress feedback to students and instructors. The effectiveness 
of time management skills training wouIq be expected td be enhanced in 
environments in v^hich instructors are appropriately trained in and 
^accepting of the training concepts. 

3. Providing students with specific study skills remediation 

which capitalizes on their active information processing strategies and/or 
provides them with methods for controlling dysfunctional behavior is 
highly feasible and promises to be ccn-effective in terms of improved 
training effectiveness and efficiency. Further enhancements to the 
effectiveness of such stud a skills training would be expected from 
efforts to refine the identification of students in need of such train- 
ing, including expanded instructor training programs and computer-based 
individualization decision models. 

4. While the Time Management module was the only product of this 
project which could be evaluated over an extended period of time, the 
results strongly suggest that benefits of specific skills training can be 
expected to increase over time. To a large extent this is probably due 
to increased instructor expertise with respect to the skills training 
materials and procedures, but a second significant component may be the 
simple acceptance of the materials and procedures as a standard part of 
the training environment. 

4 2.2 Study Skills Questionnaire . The following conclusions can 
be drawnfrom the Study Skills Questionnaire summative evaluation 
results: 

1 A self -report rating of student study skills in areas iden- 
tified'as important in a CMI environment (reading comprehension, memor- 
ization, tes. taking, concentration management) is both a reliable and 
valid method of assessing areas of student strengths and weaknesses. 
The fidelity of this measure might be further enhanced by incorporating 
instructions which stress the importance of honest answers. 

2. The Study Skills Questionnaire in its present form has suffic- 
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lent? predictive validity to be of use to CMI instructors in the pre- 
diction and/or diagnosis of those students expected to Have difficulty 
completing their course efficiently and effectively. Additional 
analyses would be required to provide instructors with the specific cut- 
off scores on each of the questionnaire's subscales that would indicate 
student need for a particular Study Skills module. 

3. J The Study Skills Questionnaire could be used as a reliable 
diagnostic and/or prescriptive tool in lieu of a battery of precourse 
assessment procedures or as a supplement to these procedures. 

4.ii.3 Instructor Orientation and Traininf| . Conclusions which can 
be drawn in this area are based both on the results of the formative 
and sunmiative evaluations of the instructor workshops, and on the 
results of the Study Skills modules evaluation. 

1. A definite requiranent exists in CllI environticnts to provide 
instructors with skills training in those roles required to effectively 
and efficiently perform their function as facilitator of the learning 
nrocess. Sur.h training can be feasibly and effectively accomplished 

in u workSMOp format which inclydes partici,)ative discussions and 
practice of nev/ skills. 

2. Instructor "training in the diagnostic and tutorial skills 
required to effectively remedy student study skills problems Is feasible 
and can be accomplished efficiently. Further enhancements to such train- 
ing might include continued instructor follow-up on the use of *:heir new 
skills and broadening the training to other areas required in a CMI 
environment (e.g., handling student motivational problems, assessing 
additional learning weaknesses). 

3. Instructor training in specific CMI roles and the skills 
required to effectively perform these roles can have a positive impact 

on instructor attitudes toward and role perceptions in a CMI environment. 

4.3 Requirements for Future Research 

This project has demonstrated the positive benefit of student skills 
training on reducing the costs of military technical training. There 
are, however, a number of questions that remain unanswered. This 
section lists these questions as areas recommended for future research. 

1. The need exists to investigate the effectiveness of the Orien- 
tation module in the AIS and/or other CHI technical .'■.raining environments 
with respect to its impact on students' performance and attitudes. 

'd. Researc' aimed at individualizing the assignment and/or re- 
assignment of time management skill training is desirable, in that it 
has the potential of further enhancing student training efficiency and 
effectiveness. 
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3. As with student training in time manafjenent skills, there is a • 
need to invesHqate methods for individual izinn tne assignment of study 
skills training in order to obtain maximum benefit from this type of ♦ 
training. 

4. Additional research which isolates the cut-off scores on the 
Stuuy Skills Questionnaire which are most reliably related to student 
pt^rformance (times, scores) and their need for particular study skills 
rui'iedidtion in CMI technical training courses, is needed to further the 
utility of this measure. 

b. A critical need exists to explore the types of roles required 
of instructors in a Cr.I environment, particularly as these relate to 
their function as facilitators of the learning process, and Specific 
training packages need to be deve-oped and evaluated in CMI training 
environments. 

4.4 Recomme ndations For Use of Materials Produce d in This Project 

1. The Orientation/Time Management Modules should be implemented 
near the beginning cf CMI technicaltraining courses to improve student 
efficiency and attitudes. 

2 The Study Skills Questionnaire should be made part of the pre- 
asscssment battery for each course or placed in^the first course block so 
that it can be used to help identify studonts with specific study skills 
problems or those who will have difficulty successfully completing the 
course. 

J The four Study Skills Modules should at least be implemented in 
all cni courses to be used by students identified as having study skills 
problems. Consideration should also be given to using these skill 
modules in non-CMI technical training courses. 

4 An Instructor Orientation and Training Workshop in those skills 
required to effectively use the Study Skills Questionnaire and Modules 
should become an on-going in-service training program in each course. 
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GLOSSARY OF TERMS 



ADAPTIVE MODEL - Consists of a set of computer programs that generate 
individual instructional assignments, predict and assign individual 
block and course completion time targets (student progress manage- 
ment), allocates training resources, and is the vehicle for 
accomplishing continual test and courseware evaluation and refinement, 

ADVANCED INSTRUCTIONAL SYSTEM (AI5) - A prototype, comprehensive computer- 
managed and computer-assisted instructional system to provide the 
following automated capabilities in support of large scale training: 
individual instructional assignments, student pi'ogress management, 
resource allocation and scheduling, information storage, and report 
generation and evaluation and re^arch. 

ALTERNATIVE MODULES - Modules utilizing different instructional approaches 
from previously existing modules to meet the specific needs of 
particular types of students and/or certain course requirements. 
(See Instructional Module). 

AUDIO-VISUAL (A/V) - A method of displaying information in which both 
the audio segment and visual segment are electro-mechanical ly and 
optically reproduced for the information of the student, e.g., 
photographic slides used in conjunction with an audio tape player. 

BLOCK - A course component comprised of lessons and modules that cover 
a specifk subject/content area and normally ends with a comprehen- 
sive tes^t. 

i 

BLOCK COMPLEflON/ELAPSED TIMES - The period of time from inception of a 
block's' first lesson or lesson group through completion of the block 



test. 

COMPUTER HARDWARE - The hardware components that comprise the computer 
system I and includes the central processor, various types of memory 
units, printout unit and control, display and distribution units. 

COMPUTER-MaUgED INSTRUCTION (CMI) - Use of the computer to manage stur 
dents (through the instructional process. The computer's role is 
that off a diagnostician and manager of instructional events. 
Through the Adaptive Model, it generates Individual Instructional 
Assignments (IIAs), predicts and assigns individual block and course 
completion times, allocates training resources and evaluates tests 
and courseware. The utilization ratio is 100 students per manage- 
ment terminal. 

COMPUTER SOFTWARE - A logical grouping of progranmed computer codes that 
gives conmands to a computer to perform a particular function. A 
unique AIS software component is the Computer Assisted/Managed 
Instructional Language (CAMIL) that facilitates both CAI and CMI. 
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GLOSSARY OF TFjmS (Continued) 



CONVENTIONAL TRAINING - Classroom and/or laboratory training conducted 
in a previously established and accepted manner, i.e., usually a 
classroom lecturer and/or laboratory instructor-student group 
relationship in a lock-step mode of progress. 

COURSE - A block or a series of blocks of instruction designed to satisfy 
Specialty Training Standards for a particular Air Force Specialty 
Code and skill level. Formal, resident training conducted at an 
Air Training Comnand installation. 

COURSE COMPLETION PREDICTIONS - A computer-generated estimate of a stu- 
dent's time required to complete the course based upon Preassessment 
Testing. Predictions may be made for completion of modules, 
lessons, or blocks as well as for the entire course. 

COURSE COMPLETION TARGET - A course completion prediction adjusted for 
course policy regarding the desired minimum, maximum and average 
course completion times. 

COURSE COMPLETION TIMES - Measured classroom time from course entry to 
graduation. 

COURSE DATA BASE - A collection of computerized files containing the 
parameters and flags which control the operation of the Adaptive 
Model for a specific course. 

COURSEWARE - Generic term for all AIS instructional material. 

CRITERION-REFERENCED TESTING - A testing methodology in which test items 
are written to assess student performance on defined behavioral 
objectives with respect to a specified standard of mastery (e.g., 70 
percent correct). 

CRITERION VARIABLES - Measures of student performance, times, and score 
on lesson and block tests. 

DAILY ROSTER - A computer generated listing of students assigned to a 
specific learning center, their assigned carrel numbers, current 
block of instruc*-ion, and rate of progress relative to their 
target course cc 'tion rate. 

EQUIVALENCE OF GROUPS - Analysis of groups of students included in the 
1ST to assure that they do not yd^y significantly in ability or 
other psychological variables (preassessment) and, as a result, do 
not bias the results of the time savings analyses. 

FIRST ATTEMPT BLOCK FAILURES - A computer produced list of the number of 
students who fail to meet criterion on their first attempt to pass 
an end-of-block test. 
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GLOSSARY OF TERMS (Continued) 



•F0RI4ATIVE EVALUATION - Th^it type of evaluation research whose purpose is 
to detect weaknesses in instructional materials or procedures and 
provide a basis^or correcting such weaknesses. 

IWOIVIDUALIZATION - See Indivi<Jualized Insthictional A£.»gnment. 

INDIVIDUALIZED INSTRUCTIONAL ASSIGNMENT - The AIS/CMI capability to 
assign individual students to alternative modules of Instruction 
for a lesson in order to achieve optimal performance from each 
student. " 

INSTRUCTOR WORKSHOPS - A training format used during the formative and 
suiTinative evaluations. Emphasis of these workshops was on the train- 
ing of instructors in study skill techniques and diagnostic, 
counseling and remediation skills, in participative discussion and 
practice sessions. 

INSTRUCTIONAL MATERIALS - Printed, audio, or visual information used in 
instruction. Includes progranmed texts, picture books, workbooks, 
audio visuals, checklists, technical orders and tests. 

INSTRUCTIONAL MODULE - A specific package of instructional materials and 
related .training resources for presentation of a specific AIS 
lesson. A lesson may have more than one instructional module. All 
modules for a lesson teach the same objectives but differ in the 
method of presentation and/or strategies used. 

INTEGRATED SYSTEM TEST - An evaluation of the Advanced Instructional 
System' designed to provide quantitative answer's regarding the 
training time reductions resulting from certain Computer Managed 
Instruction (CMI) functions of the AIS. 

INTERACTIVE (A) TERMINAL - Consists of a plasma display and keyboard and 
is used by instructors apd course authors to interact with the AIS 
central computer and da^ files and by students for on-line, 
•adaptive testing and CAI. 

LEARNING CENTERS - A learning environment to which students are assigned 
for attendance-taking purposes and in which most coursework is 
conducted. Consists of carrels, media and job related equipment 
and/or simulators designed to support individualized instruction. 

LESSON - A component of a block of instruction. Contains instructional 
information to enable achievement of a learning objective or series 
of objectives. A lesson is learned thror^h the use of one or more 
specific instructional modules. 
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GLOSSARY OF TERMS (Continued) 



MAHAGEMENT (B) TERMINAL Consists of a forms reader, printer and mini- 
computer and is used to read and score test fof^s, transmit data to 
and receive information from the AIS central computer, and print 
student prescriptions and management reports. 

MODULE - The smallest testable unit of instruction within a block of 
instruction. A set of instructional materials which applies a 
specific instructional approach for teaching a lesson. (See 
Instructional Module). 

MULII-TRACK - A concept used to describe individualization strategies 
used in alternative modules for a lesson. For example, alternative 
modules for a lesson may be produced to accommodate student needs 
by using mediums and different levels of redundancy or difficulty. 
Thus, the presentation strategies will differ for alternative 
modules in the multi -tracking category; presentation media may or 
may not differ. 

OPERATIONAL TRYOUT - Defined for the project, as a second phase of the 
formative evaluation accomplished by a large-scale evaluation. 
During this phase, the primary consideration was the effect of the 
treatment on the subjects subsequent behavior. 

PREDICTOR VARIABLES - Measures of student abilities, aptitudes, interests, 
personality, or past performance which are expected to be related to 
criterion variables of interest (e.g., student times or scores). 

PREASSESSMENT DATA - The results of a test battery given to students 
before they begin a course. The battery is designed to measure 
student abilities, attitudes, interests and backgrounds. Preassess- 
ment data, in conjunction with Within-Course Testing is used for 
Individualized Instructional Assignment and Student Progress Manage- 
ment. 

PRESCRIPTION - A computer generated student status report indicating the 
student's performance on his/her last assignment, his/her next 
assignment, and related training resources r juired, if any. 

RANDOM ASSIGNMENT - The option to specify the percentage of students who 
shou'd '2 assigned randomly to the alternative module for a lesson. 
Provides experimental control groups for AIS related research and/or 
for development of regression equations. 

RESOURCE ALLOCATION - A CMI function of the Adaptive Model for managing 
all training resources declared in the Course Data Base as computer- 
managed. The AIS capability to balance student flow through a 
module, lesson, block or course to avoid queueing as a result of 
resource unavailability and tc maximize use of critical resources. 
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SELF-PACING - A generic description of programs where learning and 

progress occur at each student's self-established pace. Generally, 
students whose rate of progress exceeds predetermined limits are 
counseled. 

SKILL TRAINING - That instructional training aimed at the remediation of 
or compensation for particular student weaknesses in academically 
related areas (e.g., self -management skills, study skills). 

SMALL-GROUP TRYOUTS - Defined in this project as the first phase of the 
formative evaluation. During this phase, the primary concern was to 
what extent the treatment was performing as designed. 

SOFTWARE - See Computer Software 

STATE OF THE ART - Current level, state, or condition of technology in 
disciplines related to- computer-based education and training. 

STRATEGIES - Specific instructional techniques applied within a module, 
lesson, block, or other sequence of instruction and designed to meet 
individual needs and characteristics of various types of students 
considering the particular learning objective. 

STUDENT DATA PROFILE (SDP) - A computerized file that maintains compre- 
hensive records for each student enrolled in an AIS course. Each 
student record contains bibliographic, prea'-.sessment and wi thin- 
course performance data. 

STUDENT PROGRESS MANAGEMENT COMPONENT (SPMC) - The AIS capability to pre- 
dict and assign individual block and course completion time targets 
based on each student's individual aptitudes, abilities, and perfor- 
mance and to provide students and instructors with daily feedback on 
each student's progress toward the target completion times. 

SUMMATIVE EVALUATION - That type of evaluation research whose purpose is 
to assess the overall operational effectiveness of a program via a 
large scale tryout. For the purposes of this project, summative 
evaluation was considered to be a second large-scale tryout in 
which data on approximately 50 students per treatment were 
collected. 

TARGETED COURSE COMPLETION RATE - A computer generated rate of progress 
through a course for each student which he/she must maintain to 
meet the course completion target. 

TEST-WISENESS - The ability to combine subject knowledge with clues in 
the test to get a score which reflects how much a person knows 
about a specific subject. 
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GLOSSARY OF TERMS (Continued) 

WITHIN COURSE DATA - Data describing student performance (lesson and 

block times and scores) within a course as compared to preassessment 
and/or bibliographic data. 
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